DOCKERFILE_ENV_FILE file starting appearing...
With dokku master, I have a DOCKERFILE_ENV_FILE file in my app when I repush my app.
If I do ps:rebuild myapp, I don't have it.
The file is created from plugins/config/docker-args-deploy
To my understanding, this file should be only created if the app is not buildstep based.
I think there is an issue here, maybe $IMAGE is empty in the case of a git push deploy master?
Hi @michaelshobbs you wrote this code, is this a bug or a feature? :)
Probably a bug. :smiley: I'll take a look.
A cursory run of `ps:rebuild` didn't yield the behavior you described. However, if somehow there was no `dokku/$APP` image then we would attempt to create this file and the `docker run` command to inspect the image would fail. Not sure what the expected behavior of `is_image_buildstep_based` should be if the image is not found. Perhaps exit with a different status?
I can't reproduce it now, weird. I updated dokku yesterday, maybe there was a fix in it, I don't know. I close the issue.
空docker 1.4 and "Driver aufs failed to remove root filesystem", "device or resource busy"
I don't think I saw this under docker 1.3.x on aufs (boot2docker on OSX), though I have with devicemapper on centos server. Got this error after upgrading docker server, retrying the `fig up` worked.
I use `brew` and my docker client is still 1.3.2. Just saw the warning in the log below, I guess that's not really supported? I'll try upgrading docker client manually.
```
$ docker version
Client version: 1.3.2
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): 39fa2fa
OS/Arch (client): darwin/amd64
Server version: 1.4.0
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 4595d4f
```
```
$ fig -f dns-fig.yml -p dns up -d
Recreating dns_hardfile_1...
Recreating dns_dnsmasq_1...
Cannot destroy container 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: Driver aufs failed to remove root filesystem 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: rename /mnt/sda1/var/lib/docker/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727 /mnt/sda1/var/lib/docker/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727-removing: device or resource busy
```
```
time="2014-12-14T20:21:27Z" level="info" msg="Container 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727 failed to exit within 10 seconds of SIGTERM - using the force"
time="2014-12-14T20:21:27Z" level="debug" msg="Sending 9 to 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727"
time="2014-12-14T20:21:28Z" level="info" msg="+job log(die, 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727, dns_dnsmasq:latest)"
time="2014-12-14T20:21:28Z" level="info" msg="-job log(die, 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727, dns_dnsmasq:latest) = OK (0)"
time="2014-12-14T20:21:28Z" level="info" msg="+job release_interface(07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727)"
...
time="2014-12-14T20:21:29Z" level="debug" msg="Calling DELETE /containers/{name:.*}"
time="2014-12-14T20:21:29Z" level="info" msg="DELETE /v1.12/containers/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727?link=False|force=False|v=False"
time="2014-12-14T20:21:29Z" level="info" msg="+job rm(07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727)"
Cannot destroy container 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: Driver aufs failed to remove root filesystem 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: r
ename /mnt/sda1/var/lib/docker/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727 /mnt/sda1/var/lib/docker/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727
-removing: device or resource busy
time="2014-12-14T20:21:29Z" level="info" msg="-job rm(07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727) = ERR (1)"
time="2014-12-14T20:21:29Z" level="error" msg="Handler for DELETE /containers/{name:.*} returned error: Cannot destroy container 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: Driver a
ufs failed to remove root filesystem 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: rename /mnt/sda1/var/lib/docker/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9
dd727 /mnt/sda1/var/lib/docker/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727-removing: device or resource busy"
time="2014-12-14T20:21:29Z" level="error" msg="HTTP Error: statusCode=500 Cannot destroy container 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: Driver aufs failed to remove root file
system 07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727: rename /mnt/sda1/var/lib/docker/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727 /mnt/sda1/var/lib/docker
/aufs/mnt/07ba11ca55a87ec6556a27007dd56a42da370ed8d78b34aed2f573266e9dd727-removing: device or resource busy"
time="2014-12-14T20:24:15Z" level="debug" msg="Calling POST /containers/{name:.*}/exec"
time="2014-12-14T20:24:15Z" level="info" msg="POST /v1.15/containers/dns_dnsmasq_1/exec"
time="2014-12-14T20:24:15Z" level="debug" msg="Warning: client and server don't have the same version (client: 1.3.2, server: 1.4.0)"
```
Still seeing this frequently on `fig up -d`, even at docker 1.4.1 on both client and server. Maybe having luck avoiding it with `--no-recreate`
```
FATA[0167] Error response from daemon: Cannot destroy container 74555d1513c4a9f3fc90ec0bb94bcf5b5ef558b6a54c5ff8a209ddd079fbf8f0: Driver aufs failed to remove root filesystem 74555d1513c4a9f3fc90ec0bb94bcf5b5ef558b6a54c5ff8a209ddd079fbf8f0: rename /mnt/sda1/var/lib/docker/aufs/mnt/74555d1513c4a9f3fc90ec0bb94bcf5b5ef558b6a54c5ff8a209ddd079fbf8f0 /mnt/sda1/var/lib/docker/aufs/mnt/74555d1513c4a9f3fc90ec0bb94bcf5b5ef558b6a54c5ff8a209ddd079fbf8f0-removing: device or resource busy
```
```
$ docker info
Containers: 28
Images: 1356
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Dirs: 1424
Execution Driver: native-0.2
Kernel Version: 3.16.7-tinycore64
Operating System: Boot2Docker 1.4.1 (TCL 5.4); master : 86f7ec8 - Tue Dec 16 23:11:29 UTC 2014
CPUs: 8
Total Memory: 3.866 GiB
Name: boot2docker
ID: 665E:OA4X:64MF:PVJF:WKXR:Q7V2:455C:HS5K:SYQW:VZIU:SI4C:HL5M
Debug mode (server): true
Debug mode (client): false
Fds: 110
Goroutines: 83
EventsListeners: 1
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
```
I'm seeing the same issue on 1.4.1.
```
$ docker info
Containers: 1
Images: 11
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Dirs: 17
Execution Driver: native-0.2
Kernel Version: 3.16.7-tinycore64
Operating System: Boot2Docker 1.4.1 (TCL 5.4); master : 86f7ec8 - Tue Dec 16 23:11:29 UTC 2014
CPUs: 8
Total Memory: 1.961 GiB
Name: boot2docker
ID: LBNP:H3GO:XVDK:GTCP:FNOX:OPY4:QJUP:QIKK:HAKK:TABZ:3NP6:4N7S
Debug mode (server): true
Debug mode(client): false
Fds: 17
Goroutines: 18
EventsListeners: 0
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
```
I'm seeing this too on Docker 1.4.1 (both client and server). But target container seems to be destroyed, at least it is not present in the `docker ps -a` output.
This may be unrelated, but just in case: I perform periodic health checks using ephemeral containers attached to the target container's network (via the `--net container:xyz` flag). It seems that one of these containers was started right before the moment target container died. As a result, this check container failed to start successfully because target network was already destroyed (andthis is totally ok).
Here is the log (I replaced ids of target and check containers with TARGET and CHECK for clarity):
``` text
//
// Creating and starting target container
// docker run -it --rm -p 10.0.42.1:4000:3000 test-app
//
time="2015-01-20T19:36:17Z" level="info" msg="POST /v1.16/containers/create"
time="2015-01-20T19:36:17Z" level="info" msg="+job create()"
time="2015-01-20T19:36:17Z" level="info" msg="+job log(create, TARGET, test-app:latest)"
time="2015-01-20T19:36:17Z" level="info" msg="-job log(create, TARGET, test-app:latest) = OK (0)"
time="2015-01-20T19:36:17Z" level="info" msg="-job create() = OK (0)"
time="2015-01-20T19:36:17Z" level="info" msg="POST /v1.16/containers/TARGET/attach?stderr=1|stdin=1|stdout=1|stream=1"
time="2015-01-20T19:36:17Z" level="info" msg="+job container_inspect(TARGET)"
time="2015-01-20T19:36:17Z" level="info" msg="-job container_inspect(TARGET) = OK (0)"
time="2015-01-20T19:36:17Z" level="info" msg="+job attach(TARGET)"
time="2015-01-20T19:36:17Z" level="info" msg="POST /v1.16/containers/TARGET/start"
time="2015-01-20T19:36:17Z" level="info" msg="+job start(TARGET)"
time="2015-01-20T19:36:17Z" level="info" msg="+job allocate_interface(TARGET)"
time="2015-01-20T19:36:17Z" level="info" msg="-job allocate_interface(TARGET) = OK (0)"
time="2015-01-20T19:36:17Z" level="info" msg="+job allocate_port(TARGET)"
time="2015-01-20T19:36:17Z" level="info" msg="-job allocate_port(TARGET) = OK (0)"
time="2015-01-20T19:36:17Z" level="info" msg="+job log(start, TARGET, test-app:latest)"
time="2015-01-20T19:36:17Z" level="info" msg="-job log(start, TARGET, test-app:latest) = OK (0)"
time="2015-01-20T19:36:17Z" level="info" msg="GET /containers/TARGET/json"
time="2015-01-20T19:36:17Z" level="info" msg="+job container_inspect(TARGET)"
time="2015-01-20T19:36:17Z" level="info" msg="-job start(TARGET) = OK (0)"
time="2015-01-20T19:36:17Z" level="info" msg="-job container_inspect(TARGET) = OK (0)"
time="2015-01-20T19:36:17Z" level="info" msg="POST /v1.16/containers/TARGET/resize?h=25|w=180"
time="2015-01-20T19:36:17Z" level="info" msg="+job resize(TARGET, 25, 180)"
time="2015-01-20T19:36:17Z" level="info" msg="-job resize(TARGET, 25, 180) = OK (0)"
// ... (unrelated log skipped)
//
// Creating and starting ephemeral check container
// docker run --name chk-N --net container:TARGET check-img some_check_command
//
time="2015-01-20T19:36:46Z" level="info" msg="POST /v1.16/containers/create?name=chk-N"
time="2015-01-20T19:36:46Z" level="info" msg="+job create(chk-N)"
time="2015-01-20T19:36:46Z" level="info" msg="+job log(create, CHECK, consul:latest)"
time="2015-01-20T19:36:46Z" level="info" msg="-job log(create, CHECK, consul:latest) = OK (0)"
time="2015-01-20T19:36:46Z" level="info" msg="-job create(chk-N) = OK (0)"
time="2015-01-20T19:36:46Z" level="info" msg="POST /v1.16/containers/CHECK/attach?stderr=1|stdout=1|stream=1"
time="2015-01-20T19:36:46Z" level="info" msg="+job container_inspect(CHECK)"
time="2015-01-20T19:36:46Z" level="info" msg="-job container_inspect(CHECK) = OK (0)"
time="2015-01-20T19:36:46Z" level="info" msg="+job attach(CHECK)"
time="2015-01-20T19:36:46Z" level="info" msg="POST /v1.16/containers/CHECK/start"
time="2015-01-20T19:36:46Z" level="info" msg="+job start(CHECK)"
time="2015-01-20T19:36:46Z" level="info" msg="+job log(start, CHECK, consul:latest)"
time="2015-01-20T19:36:46Z" level="info" msg="-job log(start, CHECK, consul:latest) = OK (0)"
//
// Target container died
//
time="2015-01-20T19:36:47Z" level="info" msg="+job log(die, TARGET, test-app:latest)"
time="2015-01-20T19:36:47Z" level="info" msg="-job log(die, TARGET, test-app:latest) = OK (0)"
time="2015-01-20T19:36:47Z" level="info" msg="+job release_interface(TARGET)"
time="2015-01-20T19:36:47Z" level="info" msg="GET /containers/CHECK/json"
time="2015-01-20T19:36:47Z" level="info" msg="+job container_inspect(CHECK)"
time="2015-01-20T19:36:47Z" level="info" msg="-job attach(TARGET) = OK (0)"
time="2015-01-20T19:36:47Z" level="info" msg="POST /v1.16/containers/TARGET/wait"
time="2015-01-20T19:36:47Z" level="info" msg="+job wait(TARGET)"
time="2015-01-20T19:36:47Z" level="info" msg="-job release_interface(TARGET) = OK (0)"
time="2015-01-20T19:36:47Z" level="info" msg="-job wait(TARGET) = OK (0)"
time="2015-01-20T19:36:47Z" level="info" msg="GET /v1.16/containers/TARGET/json"
time="2015-01-20T19:36:47Z" level="info" msg="+job container_inspect(TARGET)"
time="2015-01-20T19:36:47Z" level="info" msg="-job container_inspect(TARGET) = OK (0)"
//
// Removing target container (as a consequence of --rm cli flag)
//
time="2015-01-20T19:36:47Z" level="info" msg="DELETE /v1.16/containers/TARGET?v=1"
time="2015-01-20T19:36:47Z" level="info" msg="+job rm(TARGET)"
//
// Error removing target container: device or resource busy. This is not OK.
//
Cannot destroy container TARGET: Driver aufs failed to remove root filesystem TARGET: rename /mnt/sda1/var/lib/docker/aufs/mnt/TARGET /mnt/sda1/var/lib/docker/aufs/mnt/TARGET-removing: device or resource busy
time="2015-01-20T19:36:47Z" level="info" msg="-job rm(TARGET) = ERR (1)"
time="2015-01-20T19:36:47Z" level="error" msg="Handler for DELETE /containers/{name:.*} returned error: Cannot destroy container TARGET: Driver aufs failed to remove root filesystem TARGET: rename /mnt/sda1/var/lib/docker/aufs/mnt/TARGET /mnt/sda1/var/lib/docker/aufs/mnt/TARGET-removing: device or resource busy"
time="2015-01-20T19:36:47Z" level="error" msg="HTTP Error: statusCode=500 Cannot destroy container TARGET: Driver aufs failed to remove root filesystem TARGET: rename /mnt/sda1/var/lib/docker/aufs/mnt/TARGET /mnt/sda1/var/lib/docker/aufs/mnt/TARGET-removing: device or resource busy"
//
// Error starting check container: TARGET network is already destroyed. This is OK.
//
time="2015-01-20T19:36:47Z" level="info" msg="+job release_interface(CHECK)"
No network information to release for CHECK
time="2015-01-20T19:36:47Z" level="info" msg="-job release_interface(CHECK) = ERR (1)"
time="2015-01-20T19:36:47Z" level="info" msg="-job attach(CHECK) = OK (0)"
time="2015-01-20T19:36:47Z" level="info" msg="+job release_interface(CHECK)"
No network information to release for CHECK
time="2015-01-20T19:36:47Z" level="info" msg="-job release_interface(CHECK) = ERR (1)"
time="2015-01-20T19:36:47Z" level="info" msg="+job log(die, CHECK, consul:latest)"
time="2015-01-20T19:36:47Z" level="info" msg="-job log(die, CHECK, consul:latest) = OK (0)"
Cannot start container CHECK: setup networking failed get network namespace fd: open /proc/10959/ns/net: no such file or directory
time="2015-01-20T19:36:47Z" level="info" msg="-job start(CHECK) = ERR (1)"
time="2015-01-20T19:36:47Z" level="error" msg="Handler for POST /containers/{name:.*}/start returned error: Cannot start container CHECK: setup networking failed get network namespace fd: open /proc/10959/ns/net: no such file or directory"
time="2015-01-20T19:36:47Z" level="error" msg="HTTP Error: statusCode=404 Cannot start container CHECK: setup networking failed get network namespace fd: open /proc/10959/ns/net: no such file or directory"
```
Docker version | info:
``` text
$ docker version
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): darwin/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8
$ docker info
Containers: 5
Images: 105
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Dirs: 117
Execution Driver: native-0.2
Kernel Version: 3.16.7-tinycore64
Operating System: Boot2Docker 1.4.1 (TCL 5.4); master : 86f7ec8 - Tue Dec 16 23:11:29 UTC 2014
CPUs: 4
Total Memory: 1.961 GiB
Name: boot2docker
ID: LP33:7HWL:DFI6:5AL5:RNBL:D5ZJ:AMGD:WO3I:DGTW:SKHZ:3WFW:KNMJ
Debug mode (server): true
Debug mode (client): false
Fds: 36
Goroutines: 55
EventsListeners: 1
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
```
Same here, although not through boot2docker but on Ubuntu 14.04 directly. We have a bunch of health checks through progrium's [docker-consul](https://github.com/progrium/docker-consul) which are just curl commands that run inside a container. These containers are frequently not removed. They are created using a bash script that creates a container from within the consul container by having the `docker.sock` mounted. It's using `docker run --rm --net container:$container_id .....` as well.
The event stream shows that most health checking containers do die and destroy correctly.
```
% docker info
Containers: 3596
Images: 222
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Dirs: 7423
Execution Driver: native-0.2
Kernel Version: 3.13.0-44-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 4
Total Memory: 15.67 GiB
Name: docker02
ID: EOVE:DWGJ:AE2W:YLT5:F27M:IEJL:Q33E:FQCJ:ASIN:EHCM:ACH4:WJND
Username: bvadevops
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support
```
```
% docker version
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8
```
@remmelt, these containers are not removed because Docker ignores `--rm` flag when there is a problem launching container. This is a weird and probably undocumented, but expected behavior of Docker, so what you're seeing is in fact a `docker-consul`'s issue.
See [this comment](https://github.com/progrium/docker-consul/issues/45#issuecomment-70267512) for more info. I made [a fork](https://github.com/ficusio/docker-consul) that resolves this and several other issues of `docker-consul`. Here is [the related commit](https://github.com/ficusio/docker-consul/commit/52263479d964db902a035e419bfb98101ae30903). I'll try to get these fixes merged back to the original repo, but probably later, as I have really little time now.
The issue being discussed here occurs to me when I try to remove _target_ container (i.e. the container that health checks run against). And it is accompanied by `Driver aufs failed to remove root filesystem` error message. And, despite of an error message, the container gets removed, at least it disappears from the `docker ps -a` list.
Thank you!
I also see this error sometimes when running the Flocker test suite. For example:
```
[ERROR]
Traceback (most recent call last):
File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/twisted/trial/util.py", line 296, in _runSequentially
thing = yield d
File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/twisted/python/threadpool.py", line 196, in _worker
result = context.call(ctx, function, *args, **kwargs)
File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
File "/home/exarkun/Projects/flocker/master/flocker/node/_docker.py", line 507, in _remove
self._client.remove_container(container_name)
File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/docker/client.py", line 896, in remove_container
self._raise_for_status(res)
File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/docker/client.py", line 94, in _raise_for_status
raise errors.APIError(e, response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("Cannot destroy container DockerClientTests-flocker-node-functional-test_docker-DockerClientTests-test_list_removed_containers-888568622283900587922261: Driver aufs failed to remove root filesystem 83692922f272e460b71efdecf129064d3279f5072c2c1e70afcb38c73798c2fa: rename /var/lib/docker/aufs/diff/83692922f272e460b71efdecf129064d3279f5072c2c1e70afcb38c73798c2fa /var/lib/docker/aufs/diff/83692922f272e460b71efdecf129064d3279f5072c2c1e70afcb38c73798c2fa-removing: device or resource busy")
flocker.node.functional.test_docker.DockerClientTests.test_list_removed_containers
```
and
```
[ERROR]
Traceback (most recent call last):
File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/twisted/python/threadpool.py", line 196, in _worker
result = context.call(ctx, function, *args, **kwargs)
File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
File "/home/exarkun/Projects/flocker/master/flocker/node/_docker.py", line 507, in _remove
self._client.remove_container(container_name)
File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/docker/client.py", line 896, in remove_container
self._raise_for_status(res)
File "/home/exarkun/Environments/flocker/local/lib/python2.7/site-packages/docker/client.py", line 94, in _raise_for_status
raise errors.APIError(e, response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("Cannot destroy container flocker--912891792647--622134077079: Driver aufs failed to remove root filesystem 005d96d8ce4ca2d74242cc3873772274dff60fb5980b2ce05413d18e9d0d9d70: rename /var/lib/docker/aufs/diff/005d96d8ce4ca2d74242cc3873772274dff60fb5980b2ce05413d18e9d0d9d70 /var/lib/docker/aufs/diff/005d96d8ce4ca2d74242cc3873772274dff60fb5980b2ce05413d18e9d0d9d70-removing: device or resource busy")
flocker.node.functional.test_docker.IDockerClientNamespacedTests.test_add_and_remove
```
These are both tests that use the Docker API to create and then quickly stop and remove a container.
```
$ sudo docker version
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8
$ sudo docker info
Containers: 3
Images: 39
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Dirs: 51
Execution Driver: native-0.2
Kernel Version: 3.13.0-44-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 8
Total Memory: 62.89 GiB
Name: preon
ID: REGQ:VSJY:T7LE:7RHM:GCW7:QHGZ:K2YJ:L72Z:KLP3:54DY:JO27:EM35
WARNING: No swap limit support
```
I’m also observing this, docker 1.4.1 on Linux 3.16.0-0.bpo.4-amd64.
And some numbers: Of 1820 created short-living containers, 240 caused this error.
This is particularly annoying as `docker run` will print this error message on `stderr`, confusing tools that use the output of that to determine whether the actual job was successful.
Upgraded to 1.5.0, same problem.
Simple repro case. Under zsh, run:
while true; do echo "exit" | docker run -i --rm busybox; if (( ${?} != 0 )); then break; fi; done
Run 2-3 parallel instances of the script, if desired.
Environment:
~% docker info % 15-04-02 15:07
Containers: 222
Images: 4
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 450
Execution Driver: native-0.2
Kernel Version: 3.13.0-48-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 12
Total Memory: 23.54 GiB
Name: REDACTED
ID: REDACTED
Debug mode (server): true
Debug mode (client): false
Fds: 18
Goroutines: 26
EventsListeners: 0
Init Path: /usr/bin/docker
Docker Root Dir: /var/lib/docker
WARNING: No swap limit support
~% docker version % 15-04-02 15:07
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef
~% % 15-04-02 15:07
I tried the reproducer but unfortunately wasn't able to trigger this failure. How long do you typically run this before you see a problem?
Here's my docker info:
```
Containers: 20
Images: 231
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 271
Execution Driver: native-0.2
Kernel Version: 3.13.0-44-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 8
Total Memory: 62.89 GiB
Name: ...
ID: ...
WARNING: No swap limit support
```
and docker version
```
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef
```
At home, I can repro this in a few seconds on an Ubuntu instance under VMWare (on a Windows host). I don't have the docker info/version handy at this time.
At work, I can repro this in at most 15-20 minutes with 2 scripts running concurrently (on the configuration described above). I'll try more concurrent scripts today.
I experience this problem in ubuntu-14.04 on a vmware host
+1
Also with the Docker v1.6.0 RC5 on Ubuntu 12.04 with trusty backport kernel:
``` bash
sysadmin@slave2:~$ sudo docker info || sudo docker version
Containers: 1
Images: 805
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 807
Dirperm1 Supported: false
Execution Driver: native-0.2
Kernel Version: 3.13.0-49-generic
Operating System: Ubuntu precise (12.04.5 LTS)
CPUs: 4
Total Memory: 3.86 GiB
Name: slave2
ID: L2Y4:4XL5:J56E:SWOQ:EV7G:FBTU:4ACZ:4LBB:RAQW:FBNP:MT32:7JH7
Client version: 1.6.0-rc5
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): fc4825d
OS/Arch (client): linux/amd64
Server version: 1.6.0-rc5
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): fc4825d
OS/Arch (server): linux/amd64
```
And with the current Docker v1.5.0 on Ubuntu 12.04 with trusty backport kernel:
``` bash
sysadmin@slave2:~$ sudo docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef
sysadmin@slave2:~$
sysadmin@slave2:~$ sudo docker info
Containers: 118
Images: 1510
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 1888
Execution Driver: native-0.2
Kernel Version: 3.13.0-48-generic
Operating System: Ubuntu precise (12.04.5 LTS)
CPUs: 4
Total Memory: 3.86 GiB
Name: slave2
ID: L2Y4:4XL5:J56E:SWOQ:EV7G:FBTU:4ACZ:4LBB:RAQW:FBNP:MT32:7JH7
```
I encounter this issue as well with docker 1.4.1. It happens on a busy machine within a processing pipeline:
```
Error response from daemon: Cannot destroy container eccd3089dc18912001f2047431810e14c4b1808b470bac063a39f87bee16d5be:
Driver aufs failed to remove root filesystem eccd3089dc18912001f2047431810e14c4b1808b470bac063a39f87bee16d5be: rename /var/lib/docker/aufs/diff/eccd3089dc18912001f2047431810e14c4b1808b470bac063a39f87bee16d5be
/var/lib/docker/aufs/diff/eccd3089dc18912001f2047431810e14c4b1808b470bac063a39f87bee16d5be-removing: device or resource busy
```
+1
I am seeing the same as well with docker 1.6.0. I dont really have anything complicated running on the machine, but the containers I am trying to remove seem to be dead, if that matters.
```
sysadmin@docker10:~$ tail -F /var/logs/docker.log
time="2015-04-17T12:25:31Z" level=info msg="DELETE /v1.18/containers/b49bb2f86b12"
time="2015-04-17T12:25:31Z" level=info msg="+job rm(b49bb2f86b12)"
Cannot destroy container b49bb2f86b12: Driver aufs failed to remove root filesystem b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb: rename /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb-removing: device or resource busy
time="2015-04-17T12:25:31Z" level=info msg="-job rm(b49bb2f86b12) = ERR (1)"
time="2015-04-17T12:25:31Z" level=error msg="Handler for DELETE /containers/{name:.*} returned error: Cannot destroy container b49bb2f86b12: Driver aufs failed to remove root filesystem b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb: rename /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb-removing: device or resource busy"
time="2015-04-17T12:25:31Z" level=error msg="HTTP Error: statusCode=500 Cannot destroy container b49bb2f86b12: Driver aufs failed to remove root filesystem b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb: rename /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb-removing: device or resource busy"
Error response from daemon: Cannot destroy container b49bb2f86b12: Driver aufs failed to remove root filesystem b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb: rename /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb /var/lib/docker/aufs/diff/b49bb2f86b12163de792f9511fc4bffb49cb8b880a0a251e12591642ffb3b8eb-removing: device or resource busy
time="2015-04-17T12:25:31Z" level=info msg="DELETE /v1.18/containers/eac104ae9157"
time="2015-04-17T12:25:31Z" level=info msg="+job rm(eac104ae9157)"
Cannot destroy container eac104ae9157: Driver aufs failed to remove root filesystem eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8: rename /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8 /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8-removing: device or resource busy
time="2015-04-17T12:25:31Z" level=info msg="-job rm(eac104ae9157) = ERR (1)"
time="2015-04-17T12:25:31Z" level=error msg="Handler for DELETE /containers/{name:.*} returned error: Cannot destroy container eac104ae9157: Driver aufs failed to remove root filesystem eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8: rename /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8 /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8-removing: device or resource busy"
time="2015-04-17T12:25:31Z" level=error msg="HTTP Error: statusCode=500 Cannot destroy container eac104ae9157: Driver aufs failed to removeroot filesystem eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8: rename /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8 /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8-removing: device or resource busy"
Error response from daemon: Cannot destroy container eac104ae9157: Driver aufs failed to remove root filesystem eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8: rename /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8 /var/lib/docker/aufs/diff/eac104ae915764a2bd181fe818e0a6431d330c08f9c2369ddaede48b4b689eb8-removing: device or resource busy
```
```
sysadmin@docker10:~$ sudo docker version
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64
```
```
sysadmin@docker10:~$ sudo docker info
Containers: 3
Images: 81
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 87
Dirperm1 Supported: false
Execution Driver: native-0.2
Kernel Version: 3.10.48-3.10.6-amd64-13160847
Operating System: Ubuntu precise (12.04.2 LTS)
CPUs: 8
Total Memory: 15.68 GiB
Name: docker10
ID: CR2N:5JNE:S6XA:NNXS:FEQD:IVTM:7NCL:INCB:3AAS:CCKJ:XWUU:5FF2
```
```
sysadmin@docker10:~$ sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8aa56e3ba83f gliderlabs/registrator:latest "/bin/sh -c '/bin/re About an hour ago Up About an hour dope_container
b49bb2f86b12 hub.abc.com/application/test-img:latest "/bin/sh -c '/usr/lo 20 hours ago Dead 0.0.0.0:31000-|8080/tcp
eac104ae9157 gliderlabs/registrator:latest "/bin/sh -c '/bin/re 20 hours ago Dead
```
Albeit I'm on an older version of Ubuntu:
```
13:13:10-ubuntu~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 13.10
Release: 13.10
Codename: saucy
```
My guess is that I'd see this in newer versions as well based on the thread here.
I observed this error today as well:
```
13:05:17-ubuntu~$ sudo service docker stop
docker stop/waiting
13:06:26-ubuntu~$ sudo rm -rf /var/lib/docker
rm: cannot remove ‘/var/lib/docker/aufs’: Device or resource busy
13:06:29-ubuntu~$ ps aux | grep dockerubuntu 4313 0.0 0.0 9692 948 pts/9 S+ 13:06 0:00 grep --color=auto docker
13:06:47-ubuntu~$
```
I hit this on `docker version: 1.6.0`.
I was able to remove the container with `docker rm -f`
I originally reported this in Dec 2014 with docker 1.3.x. FWIW I've been using docker-compose and docker client|server 1.5 and 1.6 with boot2docker on osx and docker-machine, and have not seen the `Cannot destroy container` error.
I hit this on docker version: 1.6.0
the container cannot stop,just hang on it
and after restart docker service
it cannot remove
I saw this on docker 1.6.0 build 4749651 on Ubuntu 14.04. I had some containers (not sure whether they were running on killed) on docker 1.5.0. I upgraded to docker 1.6.0 and then immediately afterwards tried to remove the containers.
`docker rm -f` seemed to work
I spoke too soon, just saw this while doing a few simultaneous Dockerfile builds with docker 1.6.0 client on OSX against a digitalocean 1.6.1 server.
```
Error removing intermediate container a0568433a48a: Driver aufs failed to remove root filesystem a0568433a4...: rename /var/lib/docker/aufs/diff/a0568433a48a... /var/lib/docker/aufs/diff/a0568433a4...-removing: device or resource busy
```
`rm -f` it isn't the solution, it just remove the container entry from `ps -a`, volume still remain there, eating up precious disk space.
@gionn What about rm -vf?
@billsmith `docker rm -f -v name` doesn't work too, when the server under "heavy" load. we run all our tests with docker and when too much tests run in parallel the remove of the container fails with this error.
in case any swarm users encounter this..
I hit a variant of this with swarm just now. I restarted a 'swarm join' container on a CoreOS node that was suffering from health issues, which I suspect triggered a few container restart policies all at once.
`$ docker rm -f $id`
`Error response from daemon: 500 Internal Server Error: Cannot destroy container $id: Driver overlay failed to remove root filesystem $id: remove /var/lib/docker/overlay/$id/merged: device or resource busy
FATA[0001] Error: failed to remove one or more containers`
I waited 5 minutes, ran the same rm -f $id, and it said 'no such container', so I don't think this is a major issue. I just restarted another container with my stuff on it.
Also seeing this under Debian wheezy when running a docker-running container inside Kubernetes. docker rm -f seems to remove the container. Running Docker 1.6.
+1, http://stackoverflow.com/questions/30550472/docker-container-with-status-dead-after-consul-healthcheck-runs
no solution yet?
In the meantime I've setup a stinky workaround for this bug in the chef recipe we use to deploy containers.
These are the findings:
- it always happens on certain containers, hence I wasn't able to track down a specific characteristics e.g. volume, ports, etc (even a stupid ubuntu or centos running a sleep 9999 will trigger this, but not on a clean system, it just start to happen after a while)
- the dead container can be removed successfully (with `docker rm` without `-f` flag) after a new version of the same image is run (leading to a new dead container, but the old can now be removed)
+1 I hit this all the time, it is not consistent with any specific dockerfile
```
Error removing intermediate container 9b63e9a1e9bf: Driver aufs failed to remove root filesystem 9b63e9a1e9bf1aaa97d943b4c082eaa152e38d55bf1884bc28d88de1cc891f1a: rename /docker/aufs/mnt/9b63e9a1e9bf1aaa97d943b4c082eaa152e38d55bf1884bc28d88de1cc891f1a /docker/aufs/mnt/9b63e9a1e9bf1aaa97d943b4c082eaa152e38d55bf1884bc28d88de1cc891f1a-removing: device or resource busy
```
info
```
root@zapdos:/home/zapdos# docker -D version || docker -D info
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64
Containers: 650
Images: 8317
Storage Driver: aufs
Root Dir: /docker/aufs
Backing Filesystem: extfs
Dirs: 9626
Dirperm1 Supported: false
Execution Driver: native-0.2
Kernel Version: 3.13.0-29-generic
Operating System: Ubuntu 14.04 LTS
CPUs: 32
Total Memory: 58.81 GiB
Name: zapdos
ID: HHHH:IIII:MRNF:OYNJ:FZNT:X4EA:ZWQR:TF7G:EEEE:HSP6:RYBS:F3ER
Debug mode (server): false
Debug mode (client): true
Fds: 1111
Goroutines: 3753
System Time: Wed Jul 15 18:54:22 UTC 2015
EventsListeners: 1
Init SHA1: 11117501111dbf64c1111c27861111027c
Init Path: /usr/bin/docker
Docker Root Dir: /docker
WARNING: No swap limit support
```
@anandkumarpatel and I are seeing this very frequently with `weave attach` containers
Seeing the same on a number of containers, on various ports, with and without volumes mounted. All are linked to at least one other running container. Docker info:
```
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64
```
I got a similar issue with docker `1.8.0-dev`.
Below some useful info:
```
$ docker info
Containers: 0
Images: 18
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 18
Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-55-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 1
Total Memory: 490 MiB
Name: consul-server
ID: 22WL:EEPB:YJIQ:LA4V:UHUO:PTZ3:CQYO:F7ED:3RMS:QV7O:LURE:J4W5
WARNING: No swap limit support
Experimental: true
$ docker version
Client:
Version: 1.8.0-dev
API version: 1.20
Go version: go1.4.2
Git commit: a2ea8f2
Built: Sat Jul 25 22:03:05 UTC 2015
OS/Arch: linux/amd64
Experimental: true
Server:
Version: 1.8.0-dev
API version: 1.20
Go version: go1.4.2
Git commit: a2ea8f2
Built: Sat Jul 25 22:03:05 UTC 2015
OS/Arch: linux/amd64
Experimental: true
```
And here the description of my issue:
I had a running consul container. It was started with this command:
```
docker run -d \
-p 8500:8500 \ -p 8300-8302:8300-8302/tcp \
-p 8300-8302:8300-8302/udp \
-h consul progrium/consul -server -bootstrap
```
I tried to remove it with:
```
$ docker rm -f 5515181938cf
```
And:
```
$ docker stop 5515181938cf || docker rm -f 5515181938cf
```
And:
```
$ docker kill 5515181938cf || docker rm -f 5515181938cf
```
But got the same issue:
```
Error response from daemon: Cannot destroy container 5515181938cf: Driver aufs failed to remove root filesystem 5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076: rename /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076 /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076-removing: device or resource busy
Error: failed to remove containers: [5515181938cf]
```
Then I restarted the docker service:
```
$ service docker restart
```
After that the container was in the **Exited** state:
```
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5515181938cf progrium/consul "/bin/start -server -" About an hour ago Exited (137) 8 seconds ago 53/tcp, 0.0.0.0:8300-8302-|8300-8302/tcp, 8400/tcp, 53/udp, 0.0.0.0:8500-|8500/tcp, 0.0.0.0:8300-8302-|8300-8302/udp elegant_fermat
```
But I still couldn't remove it:
```
$ docker rm 5515181938cf
Error response from daemon: Cannot destroy container 5515181938cf: Driver aufs failed to remove root filesystem 5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076: rename /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076 /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076-removing: device or resource busy
Error: failed to remove containers: [5515181938cf]
```
Here is the content of this directory:
```
$ ls -al /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076
total 20
drwxr-xr-x 5 root root 4096 Jul 29 09:14 .
drwxr-xr-x 22 root root 4096 Jul 29 09:14 ..
drwxr-xr-x 2 root root 4096 Jul 29 09:14 data
-r--r--r-- 1 root root 0 Jul 29 09:14 .wh..wh.aufs
drwx------ 2 root root 4096 Jul 29 09:14 .wh..wh.orph
drwx------ 2 root root 4096 Jul 29 09:14 .wh..wh.plnk
```
`lsof` returned nothing:
```
$ lsof +D /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076
```
I also couldn't remove it manually:
```
$ rm -rf /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076
rm: cannot remove ‘/var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076’: Device or resource busy
```
Then I found out that the consul process was still running:
```
$ pa aux | grep consul
root 30879 0.1 2.0 33960432 10160 ? Ssl 09:14 0:09 /bin/consul agent -config-dir=/config -server -bootstrap
```
After killing the process manually I could remove the directory and the container
```
$ service docker stop
$ kill 30879
$ rm -rf /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076
$ service docker start
docker start/running, process 31775
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5515181938cf progrium/consul "/bin/start -server -" About an hour ago Dead 53/tcp, 0.0.0.0:8300-8302-|8300-8302/tcp, 0.0.0.0:8500-|8500/tcp, 0.0.0.0:8300-8302-|8300-8302/udp, 53/udp, 8400/tcp nostalgic_darwin
$ docker rm -f 5515181938cf
5515181938cf
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
```
Also seeing this with docker 1.6.0 (not the first time I've seen it).
True class: left
Text with highlighted words
Rancher-Agent crashes devicemapper with docker daemon
**Rancher Version:** 1.1.0
**Docker Version:** 1.10.3
OS and where are the hosts located? (cloud, bare metal, etc): RHEl/7.2/vmware
Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB) single node with external DB
**Environment Type:** Cattle
**Steps to Reproduce:**
Docker environment using devicemapper as storage driver with loopback device.
**Docker info:**
Server Version: 1.10.3
Storage Driver: devicemapper
Pool Name: docker-253:0-1180431-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop2
Metadata file: /dev/loop3
Data Space Used: 581 MB
Data Space Total: 107.4 GB
Data Space Available: 19.81 GB
Metadata Space Used: 1.434 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.146 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-10-14)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 3.10.0-327.4.4.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.26 GiB
**Results:**
**Containers went to unresponsive state.**
**Stopping or deleting container will end container in "dead" state.**
**Restarting docker daemon failing with below message.**
Sep 12 03:58:27 sv1lxdcim04 docker[7434]: time="2016-09-12T03:58:27.172380678-07:00" level=warning msg="devmapper: Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section."
Sep 12 03:58:27 sv1lxdcim04 docker[7434]: time="2016-09-12T03:58:27.226281019-07:00" level=error msg="[graphdriver] prior storage driver \"devicemapper\" failed: devmapper: Base Device UUID and Filesystem verification failed.devmapper: Failed to find uuid for device /dev/mapper/docker-253:2-5767170-base:exit status 2"
Sep 12 03:58:27 sv1lxdcim04 docker[7434]: time="2016-09-12T03:58:27.226350604-07:00" level=fatal msg="Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem verification failed.devmapper: Failed to find uuid for device /dev/mapper/docker-253:2-5767170-base:exit status 2"
Sep 12 03:58:27 sv1lxdcim04 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Sep 12 03:58:27 sv1lxdcim04 systemd[1]: Failed to start Docker Application Container Engine.
cadvisor process still showing as zombie process, even after docker daemon is down.
Docker daemon running back only after cleaning devicemapper directory. which clears all the image and container data
**Expected:**
空cadvisor prevents docker from removing monitored containers?
Hi all, I have a problem using cadvisor on centos 7. When cadvisor is running, docker failes to remove other containers saying that the containers filesystem is busy. After stopping cadvisor is stopped container removal is working again.
I demostrated that in this gist: https://gist.github.com/cornelius-keller/0fd2d23b68ccd88c9328
I also included os version and docker info in the gist.
Thanks for reporting, @cornelius-keller
what cadvisor version are you running? Can you get host:port/validate for cadvisor?
Is this a temporary situation, or does the container fs stays busy till you delete cadvisor?
@rjnagal
Cadvisor version is:
|pre|
[root@583274-app35 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
docker.io/google/cadvisor latest 399ae3c46a0e 47 hours ago 19.89 MB
[root@583274-app35 ~]#
|/pre|
This is a permanent situation. The container fs stays busy untill I delete cadvisor.
What do you mean by getting host:port/validate for cadvisor? Cadvisor was still running and responsive on the web ui if that is what you mean. Unfortunately I can't give you any public host port to validate as cadvisor is only exposed via a vpn.
Yeah, I just need the ouput from /validate endpoint on cadvisor UI. You can
scrub any data that's private in there. Thanks
On Fri, Jun 12, 2015 at 9:54 AM, Cornelius Keller notifications@github.com
wrote:
| @rjnagal https://github.com/rjnagal
| Cadvisor version is:
|
| [root@583274-app35 ~]# docker images| REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZEdocker.io/google/cadvisor latest 399ae3c46a0e 47 hours ago 19.89 MB
| [root@583274-app35 ~]#
|
| This is a permanent situation. The container fs stays busy untill I delete
| cadvisor.
|
| What do you mean by getting host:port/validate for cadvisor? Cadvisor was
| still running and responsive on the web ui if that is what you mean.
| Unfortunately I can't give you any public host port to validate as cadvisor
| is only exposed via a vpn.
|
| —
| Reply to this email directly or view it on GitHub
| https://github.com/google/cadvisor/issues/771#issuecomment-111555689.
Sorry was a long day, did not get that this was an endpoint. I added the output to the gist.
I am facing this same issue. Essentially, running cadvisor with `--volume=/:/rootfs:ro` causes other containers' devicemapper mounts to be mounted inside the cadvisor container, so they can't be properly destroyed when issuing `docker rm` on the target container as they will appear in use.
How can this be solved?
When i run it on Fedora 21, it works fine. But when i run it on Ubuntu 14.04.2 LTS I get the same error as described above.
Error response from daemon: Cannot destroy container xxx_jenkinsMaster_1230: Driver aufs failed to remove root filesystem 13b421d0458e740e42e5fa5ac1cb68f32638f0bc723d9ba16718955214d79b7d: rename /var/lib/docker/aufs/mnt/13b421d0458e740e42e5fa5ac1cb68f32638f0bc723d9ba16718955214d79b7d /var/lib/docker/aufs/mnt/13b421d0458e740e42e5fa5ac1cb68f32638f0bc723d9ba16718955214d79b7d-removing: device or resource busy
The main difference is, that Ubuntu uses AUFS, where Fedora uses Devicemapper. Maby thats the problem.
@rjnagal I can confirm that this issue happens on Ubuntu trusty x64 with Doceker 1.8.1, cadvisor:latest and devicemapper.
'1cb6051b30a1' being the container ID.
```
# grep -l 1cb6051b30a1 /proc/*/mountinfo
/proc/1963/mountinfo
# ps aux | grep -i 1963
root 1963 1.9 0.8 588740 71688 ? Ssl Aug26 30:08 /usr/bin/cadvisor
root 14767 0.0 0.0 11744 952 pts/0 S+ 00:56 0:00 grep --color=auto -i 1963
```
Please suggest a workaround for this.
same here with CentOS + Docker 1.8.1(devicemapper)
Had to remove `--volume=/:/rootfs:ro` || `--volume=/var/lib/docker:/var/lib/docker:ro`
@rjnagal: Excepting disk usage calculation, cAdvisor does not poke at any
of these directories right?
On Fri, Aug 28, 2015 at 12:26 AM, Jihoon Chung notifications@github.com
wrote:
| same here with CentOS + Docker 1.8.1(devicemapper)
|
| Had to remove --volume=/:/rootfs:ro ||
| --volume=/var/lib/docker:/var/lib/docker:ro
|
| —
| Reply to this email directly or view it on GitHub
| https://github.com/google/cadvisor/issues/771#issuecomment-135661164.
Same problem here with Ubuntu 14.04.3.
@difro solution works but cadvisor can't provide docker stats anymore.
Any workaround?
The last time I ran into this problem, I digged a little bit into the cAdvisor source code. I'm not 100% sure - because it was a few weeks ago - but this is essentially the gist:
If you use cAdvisor like it is shown in README.md you'll mount `/var/lib/docker` as a volume into the container. This will [create dead containers](https://github.com/docker/docker/issues/9665#issuecomment-137422559).
The reason, cAdvisor wants you to mount `/var/lib/docker` is - as far as I could see - only to display a certain info that is only interesting for admins and should be known before hand.
We should be able to get all info from a `docker inspect` rather than parsing the container config file. Seems like mounting `/var/lib/docker` is causing more trouble than it's worth.
we also encounter the same problem (`cadvisor:lastest`, `ubuntu 14.04`)
any updates regarding this?
The best we can do for now is to let users optionally disable filesystem
usage metrics. We are waiting for some of the new upstream kernel features
to simplify disk accounting.
On Tue, Jan 26, 2016 at 2:51 PM, Sven Müller notifications@github.com
wrote:
| any updates regarding this?
|
| —
| Reply to this email directly or view it on GitHub
| https://github.com/google/cadvisor/issues/771#issuecomment-175277349.
Same situation.
My Docker Version is 1.9.1
Cadvisor version 0.18.0
And when docker rm container fails, the status of that container change to "dead" .
Is it possible to umount that specific mountpoint when container status changed to "exit" or "dead" ?
+1
cAdvisor doesn't mount anything. It runs `du` periodically to collect
filesystem stats. Other than that, it does not touch the container's
filesystem at all.
The easy fix for this would be to retry docker deletion or disable
filesystem aggregation in cadvisor.
On Wed, Feb 3, 2016 at 2:57 PM, Alex Rhea notifications@github.com wrote:
| +1
|
| —
| Reply to this email directly or view it on GitHub
| https://github.com/google/cadvisor/issues/771#issuecomment-179518025.
running cAdvisor without `--volume=/:/rootfs:ro` seems to fix it.
As pointed out in https://github.com/google/cadvisor/blob/master/docs/running.md
I haven't fully tested it yet, but works fine up to now
I had to remove the following volume mounts:
- /:/rootfs:ro
- /var/lib/docker/:/var/lib/docker:ro
Setup:
- Ubuntu 14.04.3 LTS
- docker 1.9.1 with aufs
- cAdvisor 0.20.5
Upgraded docker to 1.10.3 and now cAdvisor can only see the docker images, but no containers, if I only use volume mounts:
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
If I add `/:/rootfs:ro`, cAdvisor can see the containers, but I get `device or resource busy`, when trying to remove any container.
@xbglowx Are you using the latest cadvisor release?
Using `google/cadvisor:v0.22.0`
Any ideas or suggestions how can i dig inside the issue?
cc @timstclair
I was able to reproduce this locally with docker v1.9.1 and cAdvisor 0.22.0, but only right after starting cAdvisor and only once (removing a second container works). I could not reproduce with docker v1.11.
Is this consistent with everyone else's experience?
With docker 1.11.1 the is issue is gone. With the latest fixes from docker part, seems working now.
I'm still able to reproduce this with docker 1.11.1 and cAdvisor 0.23.0. Ubuntu 14.04.
@ashkop Can you try running cAdvisor with `--disable_metrics="tcp,disk"` and see if that resolves the issue? Note that you will not get docker container filesystem metrics by adding this flag.
If I try using `--disable_metrics="tcp,disk"` I get the following:
```
sudo docker run -ti -v /var/lib/docker/:/var/lib/docker:ro -v /var/run:/var/run:rw -v /sys:/sys:ro -v /:/rootfs:ro google/cadvisor --disable_metrics="tcp,disk"
panic: assignment to entry in nil map
goroutine 1 [running]:
panic(0xb0c8c0, 0xc8201c0440)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
main.(*metricSetValue).Set(0x15ac528, 0x7ffe3cea1f59, 0x8, 0x0, 0x0)
/go/src/github.com/google/cadvisor/cadvisor.go:85 +0x1da
flag.(*FlagSet).parseOne(0xc82004e060, 0xc82005e901, 0x0, 0x0)
/usr/local/go/src/flag/flag.go:881 +0xdd9
flag.(*FlagSet).Parse(0xc82004e060, 0xc82000a100, 0x2, 0x2, 0x0, 0x0)
/usr/local/go/src/flag/flag.go:900 +0x6e
flag.Parse()
/usr/local/go/src/flag/flag.go:928 +0x6f
main.main()
/go/src/github.com/google/cadvisor/cadvisor.go:99 +0x68
```
This is with `cAdvisor version 0.23.0 (750f18e)`. Works fine with 0.22.0.
I still need to see if using `--disable_metrics="tcp,disk"` fixes the problem.
Yeah, that was fixed in https://github.com/google/cadvisor/pull/1259, but it's not integrated into any release.
@vishh Unfortunately the flag didn't help. As @xbglowx mentioned, this option causes 0.23.0 to crash, so I tried 0.22.0 and canary. Both still prevent me from removing containers. Here's the error message I get:
`Error response from daemon: Unable to remove filesystem for 9e96817fba0a443f75d1426b6d7a586f4bc84217b06eb021f6d28bae4f341473: remove /var/lib/docker/containers/9e96817fba0a443f75d1426b6d7a586f4bc84217b06eb021f6d28bae4f341473/shm: device or resource busy`
Same here on Debian 8, Docker 1.11.1 and latest cAdvisor.
@timstclair Can we make a v0.23.1 release with the fix for `--disable_metrics` flag?
I am experiencing the same issue with the following versions
"cAdvisor version: 0.23.0-750f18e"
google/cadvisor latest 5cda8139955b 8 days ago 48.92 MB
CentOS Linux release 7.2.1511 (Core)
Docker version 1.11.1, build 5604cbe
Work around was to remove /var/lib/docker from the shared volume.
@vishh Is this fixed if we just stopped tracking disk metrics for these machines? Are there other dependencies?
@rjnagal Disk metrics should be the only dependency. Disabling that by using `--disable_metrics=tcp,disk` should fix this issue.
Can we do that by default when we detect devicemapper?
@rjnagal AFAIK, it is not limited to devicemapper alone. AUFS is also affected. If we need a default solution, we will have to disable per-container disk metrics by default.
The issue persists in v0.23.1 on CentOS7, Docker 1.10.1, devicemapper
```
docker run \
--rm \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
google/cadvisor:v0.23.1 \
-docker_only \
--disable_metrics="tcp,disk"
```
To add more info - the issue persists on v0.23.1 and v0.23.2 on CentOS7, Docker 1.11.1, devicemapper.
However the issue only occurs when cadvisor is run from docker.
Running cadvisor directly on CentOS7 works without issues.
Could you add more details about your repro steps? How many containers are you running, with what options? It would help if we could reproduce from a clean VM centos image.
I tried to reproduce it on fresh VM, but failed. I'll try to find the difference that is actually causing the issue. Meanwhile I did `lsof` inside the `cadvisor` container of the file that is being blocked. Here's what I got:
```
1 /usr/bin/cadvisor pipe:[70918923]
1 /usr/bin/cadvisor pipe:[70918924]
1 /usr/bin/cadvisor pipe:[70918925]
1 /usr/bin/cadvisor socket:[70919220]
1 /usr/bin/cadvisor anon_inode:[eventpoll]
1 /usr/bin/cadvisor anon_inode:inotify
1 /usr/bin/cadvisor socket:[70919240]
```
I also noticed that issue occurs only if I start `cadvisor` after my own containers. If `cadvisor` is the first one started, then I can restart my containers without any issue.
@ashkop That's actually correct. I tried to reproduce the error, but couldn't. If the other containers are started first, only then cadvisor blocks removal.
Here's a script to replicate the error on CentOS 7.
You will need a machine with an empty block device (just replace the path to the device in `DOCKER_DATA_DISK`) and it will setup docker with devicemapper through lvm's thin-pool, run a container, then cadvisor and then stop | rm the first container.
```
#!/bin/bash
DOCKER_DATA_DISK=/dev/vdb
set -exo pipefail
setenforce Permissive
yum update -y
yum install -y lvm2
systemctl enable lvm2-lvmetad
systemctl start lvm2-lvmetadpvcreate $DOCKER_DATA_DISK
vgcreate data $DOCKER_DATA_DISK
lvcreate -l 100%free -T data/docker_thin
curl -sSL https://get.docker.com/ | sh
mkdir -p /etc/systemd/system/docker.service.d
cat ||EOF | /etc/systemd/system/docker.service.d/docker-lvm.conf
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// \
-s devicemapper \
--storage-opt dm.thinpooldev=/dev/mapper/data-docker_thin
TimeoutStartSec=3000
EOF
systemctl daemon-reload
systemctl enable docker
systemctl start docker
sleep 3
docker run \
--name=test \
-d \
debian:jessie \
/bin/sh -c "while true; do foo; sleep 1; done"
docker run \
-d \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--name=cadvisor \
google/cadvisor:v0.23.1 \
-docker_only \
--disable_metrics="tcp,disk"
docker stop test
docker rm test
```
The output is:
```
... some data ...
+ docker stop test
test
+ docker rm test
Error response from daemon: Unable to remove filesystem for 7d7513b0c3310f26e7425728f9c34e219db53a5e4dbb6e0e4259c2e6eb760044: remove /var/lib/docker/containers/7d7513b0c3310f26e7425728f9c34e219db53a5e4dbb6e0e4259c2e6eb760044/shm: device or resource busy
```
On Ubuntu 14.04, using `--disable_metrics="tcp,disk"` still does not fix the problem. I've confirmed @ashkop 's observation: If cAdvisor is started after another container, then removing said container fails.
To get around this issue i have tried running cadvisor as standalone..however it does not get data while i am using RHEL ,
cadvisor complains "unable to get fs usage from thin pool for device".. it seems it cant get right information about the storage driver.
Using RHEL 7.1
version 0.23.3 (6607e7c)
docker 1.9.1
Anybody tried similar
This issue is hitting us often and affecting production container deployments (Debian 8.5 hosts, Docker 1.11.1).
Can anyone spell out what we lose by omitting the `/:/rootfs:ro` mount? Is it just disk usage metrics?
AFAIK, it should be just the disk usage metrics
On Tue, Jul 19, 2016 at 2:38 PM, Shane StClair notifications@github.com
wrote:
| This issue is hitting us often and affecting production container
| deployments (Debian 8.5 hosts, Docker 1.11.1).
|| Can anyone spell out what we lose by omitting the /:/rootfs:ro mount? Is
| it just disk usage metrics?
|
| —
| You are receiving this because you were mentioned.
| Reply to this email directly, view it on GitHub
| https://github.com/google/cadvisor/issues/771#issuecomment-233774348,
| or mute the thread
| https://github.com/notifications/unsubscribe-auth/AGvIKN3e53lwmDwcVP7hDBloCHdfD_Dsks5qXUO_gaJpZM4FBIxe
| .
So, it is possible to `stop` cadvisor before `stop/start` any other containers and then `start` cadvisor again?
cadvisor should be the first container to start.
One should not have to worry about starting/stopping containers in order to properly run cAdvisor. Monitoring should have no affect on the running of containers.
100% agreed. I'm a telling that to workaround the issue you can start cAdvisor before other containers.
But once Cadvisor is monitoring the other containers you are not able to
remove one of the monitored ones until you remove Cadvisor, at least that
happened to me a lot until now that I stop Cadvisor update containers and
start it again. I am doing this wrong?
2016-07-24 13:25 GMT-03:00 Alex notifications@github.com:
| 100% agreed. I'm a telling that to workaround the issue you can start
| cAdvisor before other containers.
|
| —
| You are receiving this because you commented.
| Reply to this email directly, view it on GitHub
| https://github.com/google/cadvisor/issues/771#issuecomment-234786522,
| or mute the thread
| https://github.com/notifications/unsubscribe-auth/AACgalxt67a6n9U7PMSW2nyKXJRWjjP5ks5qY5IBgaJpZM4FBIxe
| .
##
Alvaro
i ran into this issue as well. removing the /:/rootfs:ro volume works around the issue for me, but i do lose some stats... looks like network inside the containers, and process lists... maybe others that i haven't noticed right off.
Docker 1.11.2, cAdvisor 0.23.1
i am able to confirm that having cAdvisor loaded first, any containers loaded after are able to be removed without the 'device or resource busy' error
Wanted to add that removing the root volume (`/:/rootfs:ro`) did **not** solve this issue for us. We ended up removing cadvisor from our deployment ecosystem until this issue is resolved as it was causing too much pain in our deployment scheme.
upgrading to Docker 1.12 also did not make any difference
Removing the root volume did not make any difference for me. Still blocking containers from removal.
docker 1.11.1
Going to remove cAdvisor from all systems as it is blocking my deployment.
We have this issue too in our production environment.
It's very frustrating because it blocks our upgrade process.
We use Debian 8, Docker 1.10.3 and cadvisor 0.23.2.
I've opted for put down cadvisor while deploying/removing containers and
then put it up again. Not liked much but works.
2016-08-03 17:09 GMT-03:00 Chad McElligott notifications@github.com:
| upgrading to Docker 1.12 also did not make any difference
|
| —
| You are receiving this because you commented.
| Reply to this email directly, view it on GitHub
| https://github.com/google/cadvisor/issues/771#issuecomment-237356597,
| or mute the thread
| https://github.com/notifications/unsubscribe-auth/AACgatIXetobsMlE0PJR7LKsnGO454m2ks5qcPWRgaJpZM4FBIxe
| .
##
Alvaro
I ended up just running cAdvisor on the host instead of as a container, and it is working well that way.
Hello Chadxz,
Did you build the latest release or the master? Because the current master branch has issues I have the impression, or I'm doing something wrong.
`../../../golang.org/x/oauth2/jws/jws.go:67:17: error: reference to undefined identifier ‘base64.RawURLEncoding’
return base64.RawURLEncoding.EncodeToString(b), nil
^
../../../golang.org/x/oauth2/jws/jws.go:85:16: error: reference to undefined identifier ‘base64.RawURLEncoding’
return base64.RawURLEncoding.EncodeToString(b), nil
^
../../../golang.org/x/oauth2/jws/jws.go:105:16: error: reference to undefined identifier ‘base64.RawURLEncoding’
return base64.RawURLEncoding.EncodeToString(b), nil
^
../../../golang.org/x/oauth2/jws/jws.go:116:25: error: reference to undefined identifier ‘base64.RawURLEncoding’
decoded, err := base64.RawURLEncoding.DecodeString(s[1])
^
`
and it goes on like that.
We too removed cadvisor from our dev systems as it created 'dead' containers when we tried to remove others.
Evert, to which tool have you moved to monitor containers?
2016-08-05 10:17 GMT-03:00 EvertMDC notifications@github.com:
| Hello Chadxz,
|
| Did you build the latest release or the master? Because the current master
| branch has issues I have the impression, or I'm doing something wrong.
|
| ../../../golang.org/x/oauth2/jws/jws.go:67:17: error: reference to
| undefined identifier ‘base64.RawURLEncoding’
| return base64.RawURLEncoding.EncodeToString(b), nil
| ^
| ../../../golang.org/x/oauth2/jws/jws.go:85:16: error: reference to
| undefined identifier ‘base64.RawURLEncoding’
| return base64.RawURLEncoding.EncodeToString(b), nil
| ^
| ../../../golang.org/x/oauth2/jws/jws.go:105:16: error: reference to
| undefined identifier ‘base64.RawURLEncoding’
| return base64.RawURLEncoding.EncodeToString(b), nil
| ^
| ../../../golang.org/x/oauth2/jws/jws.go:116:25: error: reference to
| undefined identifier ‘base64.RawURLEncoding’
| decoded, err := base64.RawURLEncoding.DecodeString(s[1])
| ^
|
| and it goes on like that.
|
| We too removed cadvisor from our dev systems as it created 'dead'
| containers when we tried to remove others.
|
| —
| You are receiving this because you commented.
| Reply to this email directly, view it on GitHub
| https://github.com/google/cadvisor/issues/771#issuecomment-237847659,
| or mute the thread
| https://github.com/notifications/unsubscribe-auth/AACgarT3UVvwHModKzo533_6tykLHwPZks5qczf_gaJpZM4FBIxe
| .
##
Alvaro
@EvertMDC I downloaded the prebuilt binary from the latest stable release on the releases page https://github.com/google/cadvisor/releases/tag/v0.23.2
Thanks @chadxz . I overlooked that.
Hello @zevarito . None at the moment but I have used the container exporter images and they worked fine. They suggested to use cadvisor however as they are no longer maintaining it.
Going to run it on the system itself now as Chadxz suggested and see how it goes.
https://github.com/docker-infra/container_exporter
Hi Evert,
I have all with Cadvisor, but I will stop monitoring the Host itself with
Cadvisor and just monitor the containers. For the Host I think Node
Exporter should be the safest bet.
2016-08-05 11:19 GMT-03:00 EvertMDC notifications@github.com:
| Thanks @chadxz https://github.com/chadxz . I overlooked that.
|
| Hello @zevarito https://github.com/zevarito . None at the moment but I
| have used the container exporter images and they worked fine. They
| suggested to use cadvisor however as they are no longer maintaining it.
| Going to run it on the system itself now as Chadxz suggested and see how
| it goes.
|
| https://github.com/docker-infra/container_exporter
|
| —
| You are receiving this because you were mentioned.
| Reply to this email directly, view it on GitHub
| https://github.com/google/cadvisor/issues/771#issuecomment-237862836,
| or mute the thread
| https://github.com/notifications/unsubscribe-auth/AACgahA5bHNx71HVm7ZWjAMOsG9OPY7sks5qc0Z3gaJpZM4FBIxe
| .
##
Alvaro
:+1: to node exporter plus cadvisor. That's what we use | very happy with that combo.
Hey @chadxz, have you continued to have success running cadvisor on the host directly?
I am facing the same problem production when running cadvisor in a container. It's hard to validate in a short period of time whether the baremetal approach will work to fix this bug when it appears so rarely.
@jaybennett89 cadvisor running on host has been working fine. The "Unable to remove filesystem" error never occurs.
True class: unrelated
Text with highlighted words
Our server was running out of disk space due to some docker diff files piling up (it seems they need to be cleaned out regularly, but that is a different topic). After making some space, `ps:rebuild` is not working anymore, it just returns immediately with no output. And we can also not push anymore to the server, we get:
```
! [remote rejected] dokku -| master (pre-receive hook declined)
error: failed to push some refs to 'dokku@.....:.....'
```
Help is greatly appreciated.
dokku report
-----| uname: Linux ubuntu-512mb-fra1-01 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
-----| memory:
total used free shared buffers cached
Mem: 994 647 346 10 19 111
-/+ buffers/cache: 515 478
Swap: 999 112 887
-----| docker version:
Client:
Version: 17.05.0-ce
API version: 1.29
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:06:06 2017
OS/Arch: linux/amd64
Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:06:06 2017
OS/Arch: linux/amd64
Experimental: false
-----| docker daemon info:
Containers: 12
Running: 5
Paused: 0
Stopped: 7
Images: 32
Server Version: 17.05.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 1133
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
apparmor
Kernel Version: 3.13.0-77-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 994MiB
Name: ubuntu-512mb-fra1-01
ID: 5XQF:UC63:3ZUV:GI2Y:JEWP:NJHA:NQ6X:5IKC:RRFA:XLTA:IILX:HBUZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): true
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
-----| sigil version: 0.4.0
-----| herokuish version:
herokuish: 0.3.31
buildpacks:
heroku-buildpack-multi v1.0.0
heroku-buildpack-ruby v163
heroku-buildpack-nodejs v99
heroku-buildpack-clojure v76
heroku-buildpack-python v109
heroku-buildpack-java v52
heroku-buildpack-gradle v22
heroku-buildpack-grails v21
heroku-buildpack-scala v76
heroku-buildpack-play v26
heroku-buildpack-php v121
heroku-buildpack-go v69
heroku-buildpack-erlang fa17af9
buildpack-nginx v8
-----| dokku version: 0.5.7
-----| dokku plugins:
plugn: 0.3.0
00_dokku-standard 0.5.7 enabled dokku core standard plugin
20_events 0.5.7 enabled dokku core events logging plugin
apps 0.5.7 enabled dokku core apps plugin
build-env 0.5.7 enabled dokku core build-env plugin
certs 0.5.7 enabled dokku core certificate management plugin
checks 0.5.7 enabled dokku core checks plugin
common 0.5.7 enabled dokku core common plugin
config 0.5.7 enabled dokku core config plugin
docker-options 0.5.7 enabled dokku core docker-options plugin
domains 0.5.7 enabled dokku core domains plugin
enter 0.5.7 enabled dokku core enter plugin
git 0.5.7 enabled dokku core git plugin
letsencrypt 0.8.6 enabled Automated installation of let's encrypt TLS certificates
logs 0.5.7 enabled dokku core logs plugin
memcached 1.0.0 enabled dokku memcached service plugin
named-containers 0.5.7 enabled dokku core named containers plugin
nginx-vhosts 0.5.7 enabled dokku core nginx-vhosts plugin
plugin 0.5.7 enabled dokku core plugin plugin
postgres 1.0.0 enabled dokku postgres service plugin
proxy 0.5.7 enabled dokku core proxy plugin
ps 0.5.7 enabled dokku core ps plugin
shell 0.5.7 enabled dokku core shell plugin storage 0.5.7 enabled dokku core storage plugin
tags 0.5.7 enabled dokku core tags plugin
tar 0.5.7 enabled dokku core tar plugin
Environment details (AWS, VirtualBox, physical, etc.): Digital Ocean
+Can you turn on trace mode and try deploying?
Okay, this is what I get when pushing:
```
+ case "$(lsb_release -si)" in
++ lsb_release -si
+ export DOKKU_DISTRO=ubuntu
+ DOKKU_DISTRO=ubuntu
+ export DOKKU_IMAGE=gliderlabs/herokuish
+ DOKKU_IMAGE=gliderlabs/herokuish
+ export DOKKU_LIB_ROOT=/var/lib/dokku
+ DOKKU_LIB_ROOT=/var/lib/dokku
+ export PLUGIN_PATH=/var/lib/dokku/plugins
+ PLUGIN_PATH=/var/lib/dokku/plugins
+ export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ export DOKKU_API_VERSION=1
+ DOKKU_API_VERSION=1
+ export DOKKU_NOT_IMPLEMENTED_EXIT=10
+ DOKKU_NOT_IMPLEMENTED_EXIT=10
+ export DOKKU_VALID_EXIT=0
+ DOKKU_VALID_EXIT=0
+ export DOKKU_LOGS_DIR=/var/log/dokku
+ DOKKU_LOGS_DIR=/var/log/dokku
+ export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ export DOKKU_CONTAINER_LABEL=dokku
+ DOKKU_CONTAINER_LABEL=dokku
+ export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ parse_args git-receive-pack ''\''myapp'\'''
+ declare 'desc=top-level cli arg parser'
+ local next_index=1
+ local skip=false
+ args=("$@")
+ local args
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=2
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=3
+ return 0
+ args=("$@")
+ [[ git-receive-pack =~ ^--.* ]]
+ has_tty
+ declare 'desc=return 0 if we have a tty'
++ /usr/bin/tty
++ true
+ [[ not a tty == \n\o\t\ \a\ \t\t\y ]]
+ return 1
+ DOKKU_QUIET_OUTPUT=1
++ id -un
+ [[ dokku != \d\o\k\k\u ]]
++ id -un
+ [[ dokku != \r\o\o\t ]]
+ [[ git-receive-pack =~ ^plugin:.* ]]
+ [[ -n git-receive-pack 'myapp' ]]
+ export -n SSH_ORIGINAL_COMMAND
+ [[ git-receive-pack =~ config-* ]]
+ [[ git-receive-pack =~ docker-options* ]]
+ set -f
+ /usr/local/bin/dokku git-receive-pack ''\''myapp'\'''
+ case "$(lsb_release -si)" in
++ lsb_release -si
+ export DOKKU_DISTRO=ubuntu
+ DOKKU_DISTRO=ubuntu
+ export DOKKU_IMAGE=gliderlabs/herokuish
+ DOKKU_IMAGE=gliderlabs/herokuish
+ export DOKKU_LIB_ROOT=/var/lib/dokku
+ DOKKU_LIB_ROOT=/var/lib/dokku
+ export PLUGIN_PATH=/var/lib/dokku/plugins
+ PLUGIN_PATH=/var/lib/dokku/plugins
+ export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ export DOKKU_API_VERSION=1
+ DOKKU_API_VERSION=1
+ export DOKKU_NOT_IMPLEMENTED_EXIT=10
+ DOKKU_NOT_IMPLEMENTED_EXIT=10
+ export DOKKU_VALID_EXIT=0
+ DOKKU_VALID_EXIT=0
+ export DOKKU_LOGS_DIR=/var/log/dokku
+ DOKKU_LOGS_DIR=/var/log/dokku
+ export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ export DOKKU_CONTAINER_LABEL=dokku
+ DOKKU_CONTAINER_LABEL=dokku
+ export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ parse_args git-receive-pack ''\''myapp'\'''
+ declare 'desc=top-level cli arg parser'
+ local next_index=1
+ local skip=false
+ args=("$@")
+ local args
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=2
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=3
+ return 0
+ args=("$@")
+ [[ git-receive-pack =~ ^--.* ]]
+ has_tty
+ declare 'desc=return 0 if we have a tty'
++ /usr/bin/tty
++ true
+ [[ not a tty == \n\o\t\ \a\ \t\t\y ]]
+ return 1
+ DOKKU_QUIET_OUTPUT=1
++ id -un
+ [[ dokku != \d\o\k\k\u ]]
++ id -un
+ [[ dokku != \r\o\o\t ]]
+ [[ git-receive-pack =~ ^plugin:.* ]]
+ [[ -n '' ]]
+ dokku_auth git-receive-pack ''\''myapp'\'''
+ declare 'desc=calls user-auth plugin trigger'
+ export SSH_USER=dokku
+ SSH_USER=dokku
+ export 'SSH_NAME=[nico]'
+ SSH_NAME='[nico]'
+ plugn trigger user-auth dokku '[nico]' git-receive-pack ''\''myapp'\'''
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ [[ ! -n '' ]]
+ return 0
+ case "$1" in
+ execute_dokku_cmd git-receive-pack ''\''myapp'\'''
+ declare 'desc=executes dokku sub-commands'
+ local PLUGIN_NAME=git-receive-pack
+ local PLUGIN_CMD=git-receive-pack
+ local implemented=0
+ local script
+ argv=("$@")
+ local argv
+ case "$PLUGIN_NAME" in
++ readlink -f /var/lib/dokku/plugins/enabled/git-receive-pack
+ [[ /var/lib/dokku/plugins/enabled/git-receive-pack == *core-plugins* ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-receive-pack/subcommands/default ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-receive-pack/subcommands/git-receive-pack ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-receive-pack/subcommands/git-receive-pack ]]
+ [[ 0 -eq 0 ]]
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/00_dokku-standard/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/20_events/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/apps/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/certs/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/checks/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/config/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/docker-options/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/domains/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/enter/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/git/commands git-receive-pack ''\''myapp'\'''
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]++ set -x
+ source /var/lib/dokku/plugins/available/apps/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++ source /var/lib/dokku/core-plugins/available/common/functions
+++ set -eo pipefail+++ [[ -n 1 ]]
+++ set -x
++ source /var/lib/dokku/plugins/available/config/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
+++ source /var/lib/dokku/core-plugins/available/common/functions
++++ set -eo pipefail
++++ [[ -n 1 ]]
++++ set -x
+ source /var/lib/dokku/plugins/available/config/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++ source /var/lib/dokku/core-plugins/available/common/functions
+++ set -eo pipefail+++ [[ -n 1 ]]
+++ set -x
+ case "$1" in
+ git_glob_cmd git-receive-pack ''\''myapp'\'''
+ declare 'desc=catch-all for any other git-* commands'
+ local 'cmd=git-*'
++ sed 's/^\///g'
++ sed 's/\\'\''/'\''/g'
++ perl -pe 's/(?|!\\)'\''//g'
++ echo ''\''myapp'\'''
+ local APP=myapp
+ local APP_PATH=/home/dokku/myapp
+ [[ git-receive-pack == \g\i\t\-\r\e\c\e\i\v\e\-\p\a\c\k ]]
+ [[ ! -d /home/dokku/myapp/refs ]]
+ [[ git-receive-pack == \g\i\t\-\r\e\c\e\i\v\e\-\p\a\c\k ]]
+ local 'args=git-receive-pack '\''/home/dokku/myapp'\'''
+ git-shell -c 'git-receive-pack '\''/home/dokku/myapp'\'''
Total 0 (delta 0), reused 0 (delta 0)
remote: + case "$(lsb_release -si)" in
remote: ++ lsb_release -si
remote: + export DOKKU_DISTRO=ubuntu
remote: + DOKKU_DISTRO=ubuntu
remote: + export DOKKU_IMAGE=gliderlabs/herokuish
remote: + DOKKU_IMAGE=gliderlabs/herokuish
remote: + export DOKKU_LIB_ROOT=/var/lib/dokku
remote: + DOKKU_LIB_ROOT=/var/lib/dokku
remote: + export PLUGIN_PATH=/var/lib/dokku/plugins
remote: + PLUGIN_PATH=/var/lib/dokku/plugins
remote: + export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
remote: + PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
remote: + export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
remote: + PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
remote: + export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
remote: + PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
remote: + export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
remote: + PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
remote: + export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
remote: + PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
remote: + export DOKKU_API_VERSION=1
remote: + DOKKU_API_VERSION=1
remote: + export DOKKU_NOT_IMPLEMENTED_EXIT=10
remote: + DOKKU_NOT_IMPLEMENTED_EXIT=10
remote: + export DOKKU_VALID_EXIT=0
remote: + DOKKU_VALID_EXIT=0
remote: + export DOKKU_LOGS_DIR=/var/log/dokku
remote: + DOKKU_LOGS_DIR=/var/log/dokku
remote: + export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
remote: + DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
remote: + export DOKKU_CONTAINER_LABEL=dokku
remote: + DOKKU_CONTAINER_LABEL=dokku
remote: + export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
remote: + DOKKU_GLOBAL_RUN_ARGS=--label=dokku
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + parse_args git-hook myapp
remote: + declare 'desc=top-level cli arg parser'
remote: + local next_index=1
remote: + local skip=false
remote: + args=("$@")
remote: + local args
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=2
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=3
remote: + return 0
remote: + args=("$@")
remote: + [[ git-hook =~ ^--.* ]]
remote: + has_tty
remote: + declare 'desc=return 0 if we have a tty'
remote: ++ /usr/bin/tty
remote: ++ true
remote: + [[ not a tty == \n\o\t\ \a\ \t\t\y ]]
remote: + return 1
remote: + DOKKU_QUIET_OUTPUT=1
remote: ++ id -un
remote: + [[ dokku != \d\o\k\k\u ]]
remote: ++ id -un
remote: + [[ dokku != \r\o\o\t ]]
remote: + [[ git-hook =~ ^plugin:.* ]]
remote: + [[ -n '' ]]
remote: + dokku_auth git-hook myapp
remote: + declare 'desc=calls user-auth plugin trigger'
remote: + export SSH_USER=dokku
remote: + SSH_USER=dokku
remote: + export 'SSH_NAME=[nico]'
remote: + SSH_NAME='[nico]'
remote: + plugn trigger user-auth dokku '[nico]' git-hook myapp
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + [[ ! -n '' ]]
remote: + return 0
remote: + case "$1" inremote: + execute_dokku_cmd git-hook myappremote: + declare 'desc=executes dokku sub-commands'
remote: + local PLUGIN_NAME=git-hook
remote: + local PLUGIN_CMD=git-hook
remote: + local implemented=0
remote: + local script
remote: + argv=("$@")
remote: + local argv
remote: + case "$PLUGIN_NAME" in
remote: ++ readlink -f /var/lib/dokku/plugins/enabled/git-hook
remote: + [[ /var/lib/dokku/plugins/enabled/git-hook == *core-plugins* ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-hook/subcommands/default ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-hook/subcommands/git-hook ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-hook/subcommands/git-hook ]]
remote: + [[ 0 -eq 0 ]]
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/00_dokku-standard/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/20_events/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/apps/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10-eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/certs/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/checks/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/config/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/docker-options/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/domains/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/enter/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/git/commands git-hook myapp
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + source /var/lib/dokku/plugins/available/apps/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: ++ source /var/lib/dokku/core-plugins/available/common/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: ++ source /var/lib/dokku/plugins/available/config/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: +++ source /var/lib/dokku/core-plugins/available/common/functions
remote: ++++ set -eo pipefail
remote: ++++ [[ -n 1 ]]
remote: ++++ set -x
remote: + source /var/lib/dokku/plugins/available/config/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: ++ source /var/lib/dokku/core-plugins/available/common/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: + case "$1" in
remote: + git_hook_cmd git-hook myapp
remote: + declare 'desc=kick off receive-app trigger from git prereceive hook'
remote: + local cmd=git-hook
remote: + local APP=myapp
remote: + local oldrev newrev refname
remote: + read -r oldrev newrev refname
remote: + [[ refs/heads/master = \r\e\f\s\/\h\e\a\d\s\/\m\a\s\t\e\r ]]
remote: + plugn trigger receive-app myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + [[ ! -n '' ]]
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + git_receive_app myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=git receive-app plugin trigger'
remote: + local trigger=git_receive_app
remote: + local APP=myapp
remote: + local REV=b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + [[ ! -d /home/dokku/myapp/refs ]]
remote: + dokku git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + case "$(lsb_release -si)" in
remote: ++ lsb_release -si
remote: + export DOKKU_DISTRO=ubuntu
remote: + DOKKU_DISTRO=ubuntu
remote: + export DOKKU_IMAGE=gliderlabs/herokuish
remote: + DOKKU_IMAGE=gliderlabs/herokuish
remote: + export DOKKU_LIB_ROOT=/var/lib/dokku
remote: + DOKKU_LIB_ROOT=/var/lib/dokku
remote: + export PLUGIN_PATH=/var/lib/dokku/plugins
remote: + PLUGIN_PATH=/var/lib/dokku/plugins
remote: + export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
remote: + PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
remote: + export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
remote: + PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
remote: + export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
remote: + PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
remote: + export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
remote: + PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
remote: + export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
remote: + PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
remote: + export DOKKU_API_VERSION=1
remote: + DOKKU_API_VERSION=1
remote: + export DOKKU_NOT_IMPLEMENTED_EXIT=10
remote: + DOKKU_NOT_IMPLEMENTED_EXIT=10
remote: + export DOKKU_VALID_EXIT=0
remote: + DOKKU_VALID_EXIT=0
remote: + export DOKKU_LOGS_DIR=/var/log/dokku
remote: + DOKKU_LOGS_DIR=/var/log/dokku
remote: + export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
remote: + DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
remote: + export DOKKU_CONTAINER_LABEL=dokku
remote: + DOKKU_CONTAINER_LABEL=dokku
remote: + export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
remote: + DOKKU_GLOBAL_RUN_ARGS=--label=dokku
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + parse_args git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=top-level cli arg parser'
remote: + local next_index=1
remote: + local skip=false
remote: + args=("$@")
remote: + local args
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=2
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=3
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=4
remote: + return 0
remote: + args=("$@")
remote: + [[ git-build =~ ^--.* ]]
remote: + has_tty
remote: + declare 'desc=return 0 if we have a tty'
remote: ++ /usr/bin/tty
remote: ++ true
remote: + [[ not a tty == \n\o\t\ \a\ \t\t\y ]]
remote: + return 1
remote: + DOKKU_QUIET_OUTPUT=1
remote: ++ id -un
remote: + [[ dokku != \d\o\k\k\u ]]
remote: ++ id -un
remote: + [[ dokku != \r\o\o\t ]]
remote: + [[ git-build =~ ^plugin:.* ]]
remote: + [[ -n '' ]]
remote: + dokku_auth git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=calls user-auth plugin trigger'
remote: + export SSH_USER=dokku
remote: + SSH_USER=dokku
remote: + export 'SSH_NAME=[nico]'
remote: + SSH_NAME='[nico]'
remote: + plugn trigger user-auth dokku '[nico]' git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + [[ ! -n '' ]]
remote: + return 0
remote: + case "$1" in
remote: + execute_dokku_cmd git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=executes dokku sub-commands'
remote: + local PLUGIN_NAME=git-build
remote: + local PLUGIN_CMD=git-build
remote: + local implemented=0
remote: + local script
remote: + argv=("$@")
remote: + local argv
remote: + case "$PLUGIN_NAME" in
remote: ++ readlink -f /var/lib/dokku/plugins/enabled/git-build
remote: + [[ /var/lib/dokku/plugins/enabled/git-build == *core-plugins* ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-build/subcommands/default ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-build/subcommands/git-build ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-build/subcommands/git-build ]]
remote: + [[ 0 -eq 0 ]]
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/00_dokku-standard/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/20_events/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/apps/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/certs/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/checks/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/config/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: +continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/docker-options/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/domains/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/enter/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/git/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + source /var/lib/dokku/plugins/available/apps/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: ++ source /var/lib/dokku/core-plugins/available/common/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: ++ source /var/lib/dokku/plugins/available/config/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: +++ source /var/lib/dokku/core-plugins/available/common/functions
remote: ++++ set -eo pipefail
remote: ++++ [[ -n 1 ]]
remote: ++++ set -x
remote: + source /var/lib/dokku/plugins/available/config/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: ++ source /var/lib/dokku/core-plugins/available/common/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: + case "$1" in
remote: + git_build_cmd git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=lock git-build'
remote: + local cmd=git-build
remote: + local APP=myapp
remote: + local APP_BUILD_LOCK=/home/dokku/myapp/.build.lock
remote: + local 'APP_BUILD_LOCK_MSG=myapp is currently being deployed or locked. Waiting...'
remote: ++ flock -n /home/dokku/myapp/.build.lock true
remote: ++ echo 0
remote: + [[ 0 -ne 0 ]]
remote: + shift 1
remote: + flock -o /home/dokku/myapp/.build.lock dokku git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + case "$(lsb_release -si)" in
remote: ++ lsb_release -si
remote: + export DOKKU_DISTRO=ubuntu
remote: + DOKKU_DISTRO=ubuntu
remote: + export DOKKU_IMAGE=gliderlabs/herokuish
remote: + DOKKU_IMAGE=gliderlabs/herokuish
remote: + export DOKKU_LIB_ROOT=/var/lib/dokku
remote: + DOKKU_LIB_ROOT=/var/lib/dokku
remote: + export PLUGIN_PATH=/var/lib/dokku/plugins
remote: + PLUGIN_PATH=/var/lib/dokku/plugins
remote: + export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
remote: + PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
remote: + export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabledremote: + PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
remote: + export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
remote: + PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
remote: + export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
remote: + PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
remote: + export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
remote: + PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
remote: + export DOKKU_API_VERSION=1
remote: + DOKKU_API_VERSION=1
remote: + export DOKKU_NOT_IMPLEMENTED_EXIT=10
remote: + DOKKU_NOT_IMPLEMENTED_EXIT=10
remote: + export DOKKU_VALID_EXIT=0remote: + DOKKU_VALID_EXIT=0
remote: + export DOKKU_LOGS_DIR=/var/log/dokku
remote: + DOKKU_LOGS_DIR=/var/log/dokku
remote: + export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
remote: + DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
remote: + export DOKKU_CONTAINER_LABEL=dokku
remote: + DOKKU_CONTAINER_LABEL=dokku
remote: + export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
remote: + DOKKU_GLOBAL_RUN_ARGS=--label=dokku
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + parse_args git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=top-level cli arg parser'
remote: + local next_index=1
remote: + local skip=false
remote: + args=("$@")
remote: + local args
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=2
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=3
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=4
remote: + return 0
remote: + args=("$@")
remote: + [[ git-build-locked =~ ^--.* ]]
remote: + has_tty
remote: + declare 'desc=return 0 if we have a tty'
remote: ++ /usr/bin/tty
remote: ++ true
remote: + [[ not a tty == \n\o\t\ \a\ \t\t\y ]]
remote: + return 1
remote: + DOKKU_QUIET_OUTPUT=1
remote: ++ id -un
remote: + [[ dokku != \d\o\k\k\u ]]
remote: ++ id -un
remote: + [[ dokku != \r\o\o\t ]]
remote: + [[ git-build-locked =~ ^plugin:.* ]]
remote: + [[ -n '' ]]
remote: + dokku_auth git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=calls user-auth plugin trigger'
remote: + export SSH_USER=dokku
remote: + SSH_USER=dokku
remote: + export 'SSH_NAME=[nico]'
remote: + SSH_NAME='[nico]'
remote: + plugn trigger user-auth dokku '[nico]' git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + [[ ! -n '' ]]
remote: + return 0
remote: + case "$1" in
remote: + execute_dokku_cmd git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=executes dokku sub-commands'
remote: + local PLUGIN_NAME=git-build-locked
remote: + local PLUGIN_CMD=git-build-locked
remote: + local implemented=0
remote: + local script
remote: + argv=("$@")
remote: + local argv
remote: + case "$PLUGIN_NAME" in
remote: ++ readlink -f /var/lib/dokku/plugins/enabled/git-build-locked
remote: + [[ /var/lib/dokku/plugins/enabled/git-build-locked == *core-plugins* ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-build-locked/subcommands/default ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-build-locked/subcommands/git-build-locked ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-build-locked/subcommands/git-build-locked ]]
remote: + [[ 0 -eq 0 ]]
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/00_dokku-standard/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/20_events/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/apps/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/certs/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/checks/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/config/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/docker-options/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/domains/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/enter/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/git/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + source /var/lib/dokku/plugins/available/apps/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: ++ source /var/lib/dokku/core-plugins/available/common/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: ++ source /var/lib/dokku/plugins/available/config/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: +++ source /var/lib/dokku/core-plugins/available/common/functions
remote: ++++ set -eo pipefail
remote: ++++ [[ -n 1 ]]
remote: ++++ set -x
remote: + source /var/lib/dokku/plugins/available/config/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: ++ source /var/lib/dokku/core-plugins/available/common/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: + case "$1" in
remote: + git_build_locked_cmd git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=setup and call git_build_app_repo'
remote: + local cmd=git-build-locked
remote: + local APP=myapp
remote: + [[ 3 -ge 3 ]]
remote: + local REF=b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + git_build_app_repo myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=builds local git app repo for app'
remote: + verify_app_name myapp
remote: + declare 'desc=verify app name format and app existence'
remote: + local APP=myapp
remote: + [[ ! -n myapp ]]
remote: + [[ ! myapp =~ ^[a-z].* ]]
remote: + [[ ! -d /home/dokku/myapp ]]
remote: + return 0
remote: + local APP=myapp
remote: + local REV=b4287d41e948d5bd423edd1d5528d86784678d8c
remote: ++ mktemp -d /tmp/dokku_git.XXXX
remote: + local GIT_BUILD_APP_REPO_TMP_WORK_DIR=/tmp/dokku_git.kXDG
remote: + trap 'rm -rf "$GIT_BUILD_APP_REPO_TMP_WORK_DIR" | /dev/null' RETURN INT TERM EXIT
remote: + local TMP_TAG=dokku/b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + chmod 755 /tmp/dokku_git.kXDG
remote: + unset GIT_DIR GIT_WORK_TREE
remote: + pushd /tmp/dokku_git.kXDG
remote: + [[ ! -d /home/dokku/myapp ]]
remote: + GIT_DIR=/home/dokku/myapp
remote: + git tag -d dokku/b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + GIT_DIR=/home/dokku/myapp
remote: + git tag dokku/b4287d41e948d5bd423edd1d5528d86784678d8c b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + git init
remote: + git config advice.detachedHead false
remote: + git remote add origin /home/dokku/myapp
remote: + git fetch --depth=1 origin refs/tags/dokku/b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + rm -rf /tmp/dokku_git.kXDG
remote: + exit_code=141
remote: + set -e
remote: + [[ 141 -eq 10 ]]
remote: + implemented=1
remote: + [[ 141 -ne 0 ]]
remote: + exit 141
remote: + exit_code=141
remote: + set -e
remote: + [[ 141 -eq 10 ]]
remote: + implemented=1
remote: + [[ 141 -ne 0 ]]
remote: + exit 141
remote: + exit_code=141
remote: + set -e
remote: + [[ 141 -eq 10 ]]
remote: + implemented=1
remote: + [[ 141 -ne 0 ]]
remote: + exit 141
+ exit_code=0
+ set -e
+ [[ 0 -eq 10 ]]
+ implemented=1
+ [[ 0 -ne 0 ]]
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/letsencrypt/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/logs/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/memcached/commands git-receive-pack ''\''myapp'\'''
+ PLUGIN_BASE_PATH=/var/lib/dokku/plugins
+ [[ -n 1 ]]
+ PLUGIN_BASE_PATH=/var/lib/dokku/plugins/enabled
+ source /var/lib/dokku/plugins/enabled/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++ dirname /var/lib/dokku/plugins/enabled/memcached/commands
+ source /var/lib/dokku/plugins/enabled/memcached/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+++ dirname /var/lib/dokku/plugins/enabled/memcached/commands
++ source /var/lib/dokku/plugins/enabled/memcached/config
+++ export MEMCACHED_IMAGE=memcached
+++ MEMCACHED_IMAGE=memcached
+++ export MEMCACHED_IMAGE_VERSION=1.4.25
+++ MEMCACHED_IMAGE_VERSION=1.4.25
+++ export MEMCACHED_ROOT=/var/lib/dokku/services/memcached
+++ MEMCACHED_ROOT=/var/lib/dokku/services/memcached
+++ export PLUGIN_COMMAND_PREFIX=memcached
+++ PLUGIN_COMMAND_PREFIX=memcached
+++ export PLUGIN_DATA_ROOT=/var/lib/dokku/services/memcached
+++ PLUGIN_DATA_ROOT=/var/lib/dokku/services/memcached
+++ PLUGIN_DATASTORE_PORTS=(11211)
+++ export PLUGIN_DATASTORE_PORTS
+++ export PLUGIN_DEFAULT_ALIAS=MEMCACHED
+++ PLUGIN_DEFAULT_ALIAS=MEMCACHED
+++ export PLUGIN_ALT_ALIAS=DOKKU_MEMCACHED
+++ PLUGIN_ALT_ALIAS=DOKKU_MEMCACHED
+++ export PLUGIN_IMAGE=memcached
+++ PLUGIN_IMAGE=memcached
+++ export PLUGIN_IMAGE_VERSION=1.4.25
+++ PLUGIN_IMAGE_VERSION=1.4.25
+++ export PLUGIN_SCHEME=memcached
+++ PLUGIN_SCHEME=memcached
+++ export PLUGIN_SERVICE=Memcached
+++ PLUGIN_SERVICE=Memcached
++ dirname /var/lib/dokku/plugins/enabled/memcached/commands
+ source /var/lib/dokku/plugins/enabled/memcached/config
++ export MEMCACHED_IMAGE=memcached
++ MEMCACHED_IMAGE=memcached
++ export MEMCACHED_IMAGE_VERSION=1.4.25
++ MEMCACHED_IMAGE_VERSION=1.4.25
++ export MEMCACHED_ROOT=/var/lib/dokku/services/memcached
++ MEMCACHED_ROOT=/var/lib/dokku/services/memcached
++ export PLUGIN_COMMAND_PREFIX=memcached
++ PLUGIN_COMMAND_PREFIX=memcached
++ export PLUGIN_DATA_ROOT=/var/lib/dokku/services/memcached
++ PLUGIN_DATA_ROOT=/var/lib/dokku/services/memcached
++ PLUGIN_DATASTORE_PORTS=(11211)
++ export PLUGIN_DATASTORE_PORTS
++ export PLUGIN_DEFAULT_ALIAS=MEMCACHED
++ PLUGIN_DEFAULT_ALIAS=MEMCACHED
++ export PLUGIN_ALT_ALIAS=DOKKU_MEMCACHED
++ PLUGIN_ALT_ALIAS=DOKKU_MEMCACHED
++ export PLUGIN_IMAGE=memcached
++ PLUGIN_IMAGE=memcached
++ export PLUGIN_IMAGE_VERSION=1.4.25
++ PLUGIN_IMAGE_VERSION=1.4.25
++ export PLUGIN_SCHEME=memcached
++ PLUGIN_SCHEME=memcached
++ export PLUGIN_SERVICE=Memcached
++ PLUGIN_SERVICE=Memcached
+ [[ git-receive-pack == memcached:* ]]
+ [[ -d /var/lib/dokku/services/memcached/* ]]
+ case "$1" in
+ exit 10
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/nginx-vhosts/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/plugin/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/postgres/commands git-receive-pack ''\''myapp'\'''
+ PLUGIN_BASE_PATH=/var/lib/dokku/plugins
+ [[ -n 1 ]]
+ PLUGIN_BASE_PATH=/var/lib/dokku/plugins/enabled
+ source /var/lib/dokku/plugins/enabled/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++ dirname /var/lib/dokku/plugins/enabled/postgres/commands
+ source /var/lib/dokku/plugins/enabled/postgres/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+++ dirname /var/lib/dokku/plugins/enabled/postgres/commands
++ source /var/lib/dokku/plugins/enabled/postgres/config
+++ export POSTGRES_IMAGE=postgres
+++ POSTGRES_IMAGE=postgres
+++ export POSTGRES_IMAGE_VERSION=9.5.0
+++ POSTGRES_IMAGE_VERSION=9.5.0
+++ export POSTGRES_ROOT=/var/lib/dokku/services/postgres
+++ POSTGRES_ROOT=/var/lib/dokku/services/postgres
+++ export PLUGIN_COMMAND_PREFIX=postgres
+++ PLUGIN_COMMAND_PREFIX=postgres
+++ export PLUGIN_DATA_ROOT=/var/lib/dokku/services/postgres
+++ PLUGIN_DATA_ROOT=/var/lib/dokku/services/postgres
+++ PLUGIN_DATASTORE_PORTS=(5432)
+++ export PLUGIN_DATASTORE_PORTS
+++ export PLUGIN_DEFAULT_ALIAS=DATABASE
+++ PLUGIN_DEFAULT_ALIAS=DATABASE
+++ export PLUGIN_ALT_ALIAS=DOKKU_POSTGRES
+++ PLUGIN_ALT_ALIAS=DOKKU_POSTGRES
+++ export PLUGIN_IMAGE=postgres
+++ PLUGIN_IMAGE=postgres
+++ export PLUGIN_IMAGE_VERSION=9.5.0
+++ PLUGIN_IMAGE_VERSION=9.5.0
+++ export PLUGIN_SCHEME=postgres
+++ PLUGIN_SCHEME=postgres
+++ export PLUGIN_SERVICE=Postgres
+++ PLUGIN_SERVICE=Postgres
++ dirname /var/lib/dokku/plugins/enabled/postgres/commands
+ source /var/lib/dokku/plugins/enabled/postgres/config
++ export POSTGRES_IMAGE=postgres
++ POSTGRES_IMAGE=postgres
++ export POSTGRES_IMAGE_VERSION=9.5.0
++ POSTGRES_IMAGE_VERSION=9.5.0
++ export POSTGRES_ROOT=/var/lib/dokku/services/postgres
++ POSTGRES_ROOT=/var/lib/dokku/services/postgres
++ export PLUGIN_COMMAND_PREFIX=postgres
++ PLUGIN_COMMAND_PREFIX=postgres
++ export PLUGIN_DATA_ROOT=/var/lib/dokku/services/postgres
++ PLUGIN_DATA_ROOT=/var/lib/dokku/services/postgres
++ PLUGIN_DATASTORE_PORTS=(5432)
++ export PLUGIN_DATASTORE_PORTS
++ export PLUGIN_DEFAULT_ALIAS=DATABASE
++ PLUGIN_DEFAULT_ALIAS=DATABASE
++ export PLUGIN_ALT_ALIAS=DOKKU_POSTGRES
++ PLUGIN_ALT_ALIAS=DOKKU_POSTGRES
++ export PLUGIN_IMAGE=postgres
++ PLUGIN_IMAGE=postgres
++ export PLUGIN_IMAGE_VERSION=9.5.0
++ PLUGIN_IMAGE_VERSION=9.5.0
++ export PLUGIN_SCHEME=postgres
++ PLUGIN_SCHEME=postgres
++ export PLUGIN_SERVICE=Postgres
++ PLUGIN_SERVICE=Postgres
+ [[ git-receive-pack == postgres:* ]]
+ [[ -d /var/lib/dokku/services/postgres/* ]]
+ case "$1" in
+ exit 10
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/proxy/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/ps/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/shell/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/storage/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/tags/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/tar/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ [[ 1 -eq 0 ]]
+ set +f
+ exit 0
```
And this when rebuilding:
```
+ case "$(lsb_release -si)" in
++ lsb_release -si
+ export DOKKU_DISTRO=ubuntu
+ DOKKU_DISTRO=ubuntu
+ export DOKKU_IMAGE=gliderlabs/herokuish
+ DOKKU_IMAGE=gliderlabs/herokuish
+ export DOKKU_LIB_ROOT=/var/lib/dokku
+ DOKKU_LIB_ROOT=/var/lib/dokku
+ export PLUGIN_PATH=/var/lib/dokku/plugins
+ PLUGIN_PATH=/var/lib/dokku/plugins
+ export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ export DOKKU_API_VERSION=1
+ DOKKU_API_VERSION=1
+ export DOKKU_NOT_IMPLEMENTED_EXIT=10
+ DOKKU_NOT_IMPLEMENTED_EXIT=10
+ export DOKKU_VALID_EXIT=0
+ DOKKU_VALID_EXIT=0
+ export DOKKU_LOGS_DIR=/var/log/dokku
+ DOKKU_LOGS_DIR=/var/log/dokku
+ export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ export DOKKU_CONTAINER_LABEL=dokku
+ DOKKU_CONTAINER_LABEL=dokku
+ export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ parse_args ps:rebuild myapp
+ declare 'desc=top-level cli arg parser'
+ local next_index=1
+ local skip=false
+ args=("$@")
+ local args
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=2
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=3
+ return 0
+ args=("$@")
+ [[ ps:rebuild =~ ^--.* ]]
+ has_tty
+ declare 'desc=return 0 if we have a tty'
++ /usr/bin/tty
+ [[ /dev/pts/0 == \n\o\t\ \a\ \t\t\y ]]
+ return 0
++ id -un
+ [[ root != \d\o\k\k\u ]]
+ [[ ! ps:rebuild =~ plugin:* ]]
++ id -un
+ export SSH_USER=root
+ SSH_USER=root
+ sudo -u dokku -E -H /usr/local/bin/dokku ps:rebuild myapp
+ case "$(lsb_release -si)" in
++ lsb_release -si
+ export DOKKU_DISTRO=ubuntu
+ DOKKU_DISTRO=ubuntu
+ export DOKKU_IMAGE=gliderlabs/herokuish
+ DOKKU_IMAGE=gliderlabs/herokuish
+ export DOKKU_LIB_ROOT=/var/lib/dokku
+ DOKKU_LIB_ROOT=/var/lib/dokku
+ export PLUGIN_PATH=/var/lib/dokku/plugins
+ PLUGIN_PATH=/var/lib/dokku/plugins
+ export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ export DOKKU_API_VERSION=1
+ DOKKU_API_VERSION=1
+ export DOKKU_NOT_IMPLEMENTED_EXIT=10
+ DOKKU_NOT_IMPLEMENTED_EXIT=10
+ export DOKKU_VALID_EXIT=0
+ DOKKU_VALID_EXIT=0
+ export DOKKU_LOGS_DIR=/var/log/dokku
+ DOKKU_LOGS_DIR=/var/log/dokku
+ export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ export DOKKU_CONTAINER_LABEL=dokku
+ DOKKU_CONTAINER_LABEL=dokku
+ export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ parse_args ps:rebuild myapp
+ declare 'desc=top-level cli arg parser'
+ local next_index=1
+ local skip=false
+ args=("$@")
+ local args
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=2
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=3
+ return 0
+ args=("$@")
+ [[ ps:rebuild =~ ^--.* ]]
+ has_tty
+ declare 'desc=return 0 if we have a tty'
++ /usr/bin/tty
+ [[ /dev/pts/0 == \n\o\t\ \a\ \t\t\y ]]
+ return 0
++ id -un
+ [[ dokku != \d\o\k\k\u ]]
++ id -un
+ [[ dokku != \r\o\o\t ]]
+ [[ ps:rebuild =~ ^plugin:.* ]]
+ [[ -n '' ]]
+ dokku_auth ps:rebuild myapp
+ declare 'desc=calls user-auth plugin trigger'
+ export SSH_USER=root
+ SSH_USER=root
+ export SSH_NAME=default
+ SSH_NAME=default
+ plugn trigger user-auth root default ps:rebuild myapp
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ [[ ! -n '' ]]
+ return 0
+ case "$1" in
+ execute_dokku_cmd ps:rebuild myapp
+ declare 'desc=executes dokku sub-commands'+ local PLUGIN_NAME=ps:rebuild
+ local PLUGIN_CMD=ps:rebuild
+ local implemented=0
+ local script
+ argv=("$@")
+ local argv
+ case "$PLUGIN_NAME" in
++ readlink -f /var/lib/dokku/plugins/enabled/ps
+ [[ /var/lib/dokku/core-plugins/available/ps == *core-plugins* ]]
+ [[ ps:rebuild == \p\s\:\r\e\b\u\i\l\d ]]
+ shift 1
+ [[ ! -z '' ]]
+ set -- ps:rebuild myapp
+ [[ -x /var/lib/dokku/plugins/enabled/ps:rebuild/subcommands/default ]]
+ [[ -x /var/lib/dokku/plugins/enabled/ps:rebuild/subcommands/ps:rebuild ]]
+ [[ -x /var/lib/dokku/plugins/enabled/ps/subcommands/rebuild ]]
+ /var/lib/dokku/plugins/enabled/ps/subcommands/rebuild ps:rebuild myapp
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ source /var/lib/dokku/plugins/available/ps/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x++ source /var/lib/dokku/core-plugins/available/common/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
++ source /var/lib/dokku/plugins/available/config/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
+++ source /var/lib/dokku/core-plugins/available/common/functions
++++ set -eo pipefail
++++ [[ -n 1 ]]
++++ set -x
+ ps_rebuild_cmd ps:rebuild myapp
+ declare 'desc=rebuilds app via command line'
+ local cmd=ps:rebuild
+ [[ -z myapp ]]
+ ps_rebuild myapp
+ declare 'desc=rebuilds app from base image'
+ local APP=myapp
+ verify_app_name myapp
+ declare 'desc=verify app name format and app existence'
+ local APP=myapp
+ [[ ! -n myapp ]]
+ [[ ! myapp =~ ^[a-z].* ]]
+ [[ ! -d /home/dokku/myapp ]]
+ return 0
+ plugn trigger receive-app myapp
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ [[ ! -n '' ]]
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ git_receive_app myapp
+ declare 'desc=git receive-app plugin trigger'
+ local trigger=git_receive_app
+ local APP=myapp
+ local REV=
+ [[ ! -d /home/dokku/myapp/refs ]]
+ dokku git-build myapp
+ case "$(lsb_release -si)" in
++ lsb_release -si
+ export DOKKU_DISTRO=ubuntu
+ DOKKU_DISTRO=ubuntu
+ export DOKKU_IMAGE=gliderlabs/herokuish
+ DOKKU_IMAGE=gliderlabs/herokuish
+ export DOKKU_LIB_ROOT=/var/lib/dokku
+ DOKKU_LIB_ROOT=/var/lib/dokku
+ export PLUGIN_PATH=/var/lib/dokku/plugins
+ PLUGIN_PATH=/var/lib/dokku/plugins
+ export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ export DOKKU_API_VERSION=1
+ DOKKU_API_VERSION=1
+ export DOKKU_NOT_IMPLEMENTED_EXIT=10
+ DOKKU_NOT_IMPLEMENTED_EXIT=10
+ export DOKKU_VALID_EXIT=0
+ DOKKU_VALID_EXIT=0
+ export DOKKU_LOGS_DIR=/var/log/dokku
+ DOKKU_LOGS_DIR=/var/log/dokku
+ export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ export DOKKU_CONTAINER_LABEL=dokku
+ DOKKU_CONTAINER_LABEL=dokku
+ export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ parse_args git-build myapp
+ declare 'desc=top-level cli arg parser'
+ local next_index=1
+ local skip=false
+ args=("$@")
+ local args
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=2
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=3
+ return 0
+ args=("$@")
+ [[ git-build =~ ^--.* ]]
+ has_tty
+ declare 'desc=return 0 if we have a tty'
++ /usr/bin/tty
+ [[ /dev/pts/0 == \n\o\t\ \a\ \t\t\y ]]
+ return 0
++ id -un
+ [[ dokku != \d\o\k\k\u ]]
++ id -un
+ [[ dokku != \r\o\o\t ]]
+ [[ git-build =~ ^plugin:.* ]]
+ [[ -n '' ]]
+ dokku_auth git-build myapp
+ declare 'desc=calls user-auth plugin trigger'
+ export SSH_USER=root
+ SSH_USER=root
+ export SSH_NAME=default
+ SSH_NAME=default
+ plugn trigger user-auth root default git-build myapp
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ [[ ! -n '' ]]
+ return 0
+ case "$1" in
+ execute_dokku_cmd git-build myapp
+ declare 'desc=executes dokku sub-commands'
+ local PLUGIN_NAME=git-build
+ local PLUGIN_CMD=git-build
+ local implemented=0
+ local script
+ argv=("$@")
+ local argv
+ case "$PLUGIN_NAME" in
++ readlink -f /var/lib/dokku/plugins/enabled/git-build
+ [[ /var/lib/dokku/plugins/enabled/git-build == *core-plugins* ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-build/subcommands/default ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-build/subcommands/git-build ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-build/subcommands/git-build ]]
+ [[ 0 -eq 0 ]]
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/00_dokku-standard/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/20_events/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/apps/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/certs/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/checks/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/config/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/docker-options/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/domains/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/enter/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/git/commands git-build myapp
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ source /var/lib/dokku/plugins/available/apps/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++ source /var/lib/dokku/core-plugins/available/common/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
++ source /var/lib/dokku/plugins/available/config/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
+++ source /var/lib/dokku/core-plugins/available/common/functions
++++ set -eo pipefail
++++ [[ -n 1 ]]
++++ set -x
+ source /var/lib/dokku/plugins/available/config/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++ source /var/lib/dokku/core-plugins/available/common/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
+ case "$1" in
+ git_build_cmd git-build myapp
+ declare 'desc=lock git-build'
+ local cmd=git-build
+ local APP=myapp
+ local APP_BUILD_LOCK=/home/dokku/myapp/.build.lock
+ local 'APP_BUILD_LOCK_MSG=myapp is currently being deployed or locked. Waiting...'
++ flock -n /home/dokku/myapp/.build.lock true
++ echo 0
+ [[ 0 -ne 0 ]]
+ shift 1
+ flock -o /home/dokku/myapp/.build.lock dokku git-build-locked myapp
+ case "$(lsb_release -si)" in
++ lsb_release -si
+ export DOKKU_DISTRO=ubuntu
+ DOKKU_DISTRO=ubuntu
+ export DOKKU_IMAGE=gliderlabs/herokuish
+ DOKKU_IMAGE=gliderlabs/herokuish
+ export DOKKU_LIB_ROOT=/var/lib/dokku
+ DOKKU_LIB_ROOT=/var/lib/dokku
+ export PLUGIN_PATH=/var/lib/dokku/plugins
+ PLUGIN_PATH=/var/lib/dokku/plugins
+ export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ export DOKKU_API_VERSION=1
+ DOKKU_API_VERSION=1
+ export DOKKU_NOT_IMPLEMENTED_EXIT=10
+ DOKKU_NOT_IMPLEMENTED_EXIT=10
+ export DOKKU_VALID_EXIT=0
+ DOKKU_VALID_EXIT=0
+ export DOKKU_LOGS_DIR=/var/log/dokku
+ DOKKU_LOGS_DIR=/var/log/dokku
+ export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ export DOKKU_CONTAINER_LABEL=dokku
+ DOKKU_CONTAINER_LABEL=dokku
+ export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ parse_args git-build-locked myapp
+ declare 'desc=top-level cli arg parser'
+ local next_index=1
+ local skip=false
+ args=("$@")
+ local args
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=2
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=3
+ return 0
+ args=("$@")
+ [[ git-build-locked =~ ^--.* ]]
+ has_tty
+ declare 'desc=return 0 if we have a tty'
++ /usr/bin/tty
+ [[ /dev/pts/0 == \n\o\t\ \a\ \t\t\y ]]
+ return 0
++ id -un
+ [[ dokku != \d\o\k\k\u ]]
++ id -un
+ [[ dokku != \r\o\o\t ]]
+ [[ git-build-locked =~ ^plugin:.* ]]
+ [[ -n '' ]]
+ dokku_auth git-build-locked myapp
+ declare 'desc=calls user-auth plugin trigger'
+ export SSH_USER=root
+ SSH_USER=root
+ export SSH_NAME=default
+ SSH_NAME=default
+ plugn trigger user-auth root default git-build-locked myapp
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ [[ ! -n '' ]]
+ return 0
+ case "$1" in
+ execute_dokku_cmd git-build-locked myapp
+ declare 'desc=executes dokku sub-commands'
+ local PLUGIN_NAME=git-build-locked
+ local PLUGIN_CMD=git-build-locked
+ local implemented=0
+ local script
+ argv=("$@")
+ local argv
+ case "$PLUGIN_NAME" in
++ readlink -f /var/lib/dokku/plugins/enabled/git-build-locked
+ [[ /var/lib/dokku/plugins/enabled/git-build-locked == *core-plugins* ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-build-locked/subcommands/default ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-build-locked/subcommands/git-build-locked ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-build-locked/subcommands/git-build-locked ]]
+ [[ 0 -eq 0 ]]
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/00_dokku-standard/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/20_events/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/apps/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/certs/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/checks/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/config/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/docker-options/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/domains/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/enter/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
```
Can you remove the redirect to stderr on [this line](https://github.com/dokku/dokku/blob/a9a9d0a898c23a1df9252b6ae1f8fecc2ff1be4e/plugins/git/functions#L57) and try pushing again?
Ok, I get this:
```
remote: + git fetch --depth=1 origin refs/tags/dokku/b4287d41e948d5bd423edd1d5528d86784678d8c
remote: fatal: write error: No space left on device
```
But I have a different version of that git file you linked to, it is not called `functions` but `commands` (`dokku/plugins/enabled/git/commands`).
Okay, so even though I cleaned up there is no space. Hm. Any idea how I can make more space? Most of the space is taken in /var/lib/docker/aufs/diff.
It is a 30 GB server and there is only one app. :)
Here is the disk space info if that helps:
```Filesystem Size Used Avail Use% Mounted on
udev 487M 4.0K 487M 1% /dev
tmpfs 100M 552K 99M 1% /run
/dev/disk/by-label/DOROOT 30G 27G1.4G 96% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 497M 2.8M 495M 1% /run/shm
none 100M 0 100M 0% /run/user
overflow 1.0M 0 1.0M 0% /tmp
none 30G 27G 1.4G 96% /var/lib/docker/aufs/mnt/6cf7122027c0fc8f30350803dc83d016398b69bf17e5b69349b8559718d787f6
shm 64M 0 64M 0% /var/lib/docker/containers/5d0b9c824ea6a7b05c73293f5ac92e8bc6060cd9b69d48aef2a90555a6124a4c/shm
none 30G 27G 1.4G 96% /var/lib/docker/aufs/mnt/96c252a39a50a42b7f30b641c4c35bf3fe4fc339990a7127d7a7f6a83aad4e75
shm 64M 0 64M 0% /var/lib/docker/containers/f149b478793abf788bb3c13b89509965e678083db541cc7a762059dcbcdb2737/shm
none 30G 27G 1.4G 96% /var/lib/docker/aufs/mnt/6a54418da4a675d48752d2bfc679ea3d72b30fd744732450d3208931edfea689
shm 64M 4.0K 64M 1% /var/lib/docker/containers/0e442500e4008b22e8247296a9a79e67fe4aeac3526dc265fa1fa633e77ef77b/shm
none 30G 27G 1.4G 96% /var/lib/docker/aufs/mnt/947a39aeaadd4774c45360bb6bd792ca94cbae4570e29147fad213e188137e70
shm 64M 0 64M 0% /var/lib/docker/containers/64af2fc33af08a1e7907d5ff2f419a5187c8c170eace4f6a08a65ac6484d3e53/shm
none 30G 27G 1.4G 96% /var/lib/docker/aufs/mnt/163a716c9ff437e0dc48aea195b5907cb78e820aa4838a65efb7c8849a66e361
shm 64M 0 64M 0% /var/lib/docker/containers/3cb07b2cd8962f3c9e27d032e3bdd033bcfb4e723f943a5cc2193907ba662919/shm
```
I think my issue is this docker issue: https://github.com/moby/moby/issues/22207
Okay, was running one of the scripts suggested in the scripts which broke my build, but made space, so after a rebuild (which worked again), the app started up. I was also able to push updates again. However the server disk is still very full (90%). Wonder how to properly solve this. Seems there are still over 20GB unused files.
My guess is a restart will clear out any files that are still there lazily - like log files. Once you restart, just `ps:rebuild` your app and call it a day?
I did a rebuild, unfortunately the /var/lib/docker/aufs/diff is still huge I need to solve this. Server is almost full again. But not sure if it has to do with dokku at all or if it is a docker issue. At least i updated docker, dokku and nginx now. There is a script that seems to fix this temporarily: https://gist.github.com/Karreg/84206b9711cbc6d0fbbe77a57f705979 And it is also suggested that the latest docker version doesn't have the issue anymore. However, that script will delete all volumes, containers and images. What would I need to do to get my dokku app running again after I run it?
In theory we persist all data to disk, so unless you are using random volumes, you *should* be fine to run it and then `ps:rebuild` your application. The `ps:rebuild` will pull down a new herokuish - or any other stuff you need for the build.
You could also try upgrading docker, which maybe will fix it automatically according to comments in the issue.
Note: this isn't a Dokku issue, but a docker one. We can't do anything about diff files laying around, so unless thats the new normal - great! - then I'm going to leave it as is.
Ran the script but now the memcached container is not found when doing the rebuild. Do I have to set it up separately?
Can you hop on the slack #dokku for support?
I have slack with our own workspace, how can I get into #dokku?
```
dokku memcached:list
Error: No such object: dokku.memcached.name
Error: No such object: 5d0b9c824ea6a7b05c73293f5ac92e8bc6060cd9b69d48aef2a90555a6124a4c
Error: No such object: 5d0b9c824ea6a7b05c73293f5ac92e8bc6060cd9b69d48aef2a90555a6124a4c
Error: No such object: 5d0b9c824ea6a7b05c73293f5ac92e8bc6060cd9b69d48aef2a90555a6124a4c
Error: No such object: 5d0b9c824ea6a7b05c73293f5ac92e8bc6060cd9b69d48aef2a90555a6124a4c
Error: No such object: 5d0b9c824ea6a7b05c73293f5ac92e8bc6060cd9b69d48aef2a90555a6124a4c
NAME VERSION STATUS EXPOSED PORTS LINKS
name stopped - app-name
```
Same with postgres.
http://dokku.viewdocs.io/dokku/getting-started/where-to-get-help/
do i need to create new services? hope i didnt lose my db?
Please jump on our slack chat to get support, github issues isn't the best method for discussing your particular issue. I'll update the ticket once we've gone through everything.
Using postgres as an example, if you ever have to recover from suddenly deleting/losing all your containers/images/volumes, fear not! Dokku stores the data on disk, so as long as you aren't mucking about in your `/var/lib/dokku` directory, the following will get your services running again:
```shell
# note what version of the service is running (in this case, 9.5.9 for postgres)
# update the plugin so we get any necessary fixes!
dokku plugin:update postgres
# stop the running version
dokku postgres:stop SERVICE_NAME
# export the correct image version or you'll be auto-updated to the latest, which might result in an error depending on your database
export POSTGRES_IMAGE_VERSION=9.5.9
# start the service
dokku postgres:start SERVICE_NAME
```
Once your services are running, a `ps:rebuild` will properly rebuild your applications.
Sorry to comment on a close thread. I had the exact same problem as @ncri. _git push_ was still failing and complaining about no disk space remaining even though there were plenty once I deleted unused containers. Restarting the docker service didn't help.
Simply rebooting my server (sudo reboot) fixed the issue for me.
`dokku cleanup` command does not work for me either, no idea why. I have to run the docker commands every time I run out of space
@gabrielhpugliese Without knowing more about your setup, debugging this is going to be hard. Please file a new issue and include all the information we ask for in the issue template.空React version: 17.0.0.rc2
## Steps To Reproduce
1. Install in dev mode
2. all events, click, onBlur all of them are too slowly every time you click in a input or something freeze the browser
And even more if you open the chrome dev tools or others becomes more slow+It doesn't look like this bug report has enough info for one of us to reproduce it.
Please provide a CodeSandbox (https://codesandbox.io/s/new), a link to a repository on GitHub, or provide a minimal code example that reproduces the problem. Screenshots or videos can also be helpful if they help provide context on how to repro the bug.
Here are some tips for providing a minimal example: https://stackoverflow.com/help/mcve
cc @trueadm
@joacub Are there any DEV warnings in the console during this period of slowness?
| @joacub Are there any DEV warnings in the console during this period of slowness?
No there is no warnings, i do a js profiler and react call severals times to one same function over and over, i will share this soon
this is related
https://github.com/facebook/react/issues/19958
here is the profiler:
|img width="1250" alt="Screen Shot 2020-10-05 at 2 07 28 PM" src="https://user-images.githubusercontent.com/2091228/95115875-2a93c600-0714-11eb-8bb8-1dcbb2eadf3e.png"||img width="1257" alt="Screen Shot 2020-10-05 at 2 06 53 PM" src="https://user-images.githubusercontent.com/2091228/95115884-2d8eb680-0714-11eb-812a-932271847e0d.png"|
and there is more and more calls to the same function and the other function discreteUpdates too take long:
|img width="1255" alt="Screen Shot 2020-10-05 at 2 09 07 PM" src="https://user-images.githubusercontent.com/2091228/95116009-5e6eeb80-0714-11eb-902c-ce62bb40f4d2.png"|
That is a recursive function that React calls once per component withmutation effects (e.g. add `|div|` to the DOM).
| this is related
#19958
Possibly, though hard to say with confidence yet.
| That is a recursive function that React calls once per component with mutation effects (e.g. add `|div|` to the DOM).
|
| | this is related
| | #19958
|
| Possibly, though hard to say with confidence yet.
like 1000 times ? what do you mean mutation effects? this ---| **document.appendChild** or what ?
this works perfectly in production and in safari and when the dev tools is close works better but still slowly.
im even more im sure im not using mutation effect like **document.appendChild** if yo are talking about that.
@joacub Brian is saying it is absolutely *expected* to see that function in the stack trace a lot — basically, once per `|div|` or a component that you're mounting. What would not be expected is seeing it more times than the nesting of your tree. It's hard to say anything from your screenshots — it would really help if you could either isolate the problematic behavior in an example or share your project or deploy it somewhere.
| @joacub Brian is saying it is absolutely _expected_ to see that function in the stack trace a lot — basically, once per `|div|` or a component that you're mounting. What would not be expected is seeing it more times than the nesting of your tree. It's hard to say anything from your screenshots — it would really help if you could either isolate the problematic behavior in an example or share your project or deploy it somewhere.
project is private so I can't share, but I understand what he said, im not remounting all tree is just one button nothing else, that's happens in all click events, all of my components are using React.memo with () =| true always so never has to be remounted or called. and more that click is in one separate element which is in outside of my tree child components.
before react 17 this was working pretty good now is unstable 100%, I can't even work
A little more context: Even if only one component in your tree has a mutation effect, React will recurse from the root down to that component.
```
A
B C
D E
F G
```
Let's say G has a mutation effect, React would still recurse from A -| B -| D -| G to apply the effect. The deeper your tree is, the taller the call stack would be.
| before react 17 this was working pretty good now is unstable 100%, I can't even work
I don't know what this means.
It's very difficult for us to guess about what might be causing this issue without a repro. If you can't share your private app with us, can you make a reduced case that shows the problem on Code Sandbox?
|project is private so I can't share
Is it deployed somewhere in production that we could access? E.g. maybe it's a live website.
| | project is private so I can't share
|
| Is it deployed somewhere in production that we could access? E.g. maybe it's a live website.
yes but production works perfect, just in dev mode is super slow.
you can access here bringerparcel.dev
| A little more context:Even if only one component in your tree has a mutation effect, React will recurse from the root down to that component.
|
| ```
| A
| B C
| D E
| F G
| ```
|
| Let's say G has a mutation effect, React would still recurse from A -| B -| D -| G to apply the effect. The deeper your tree is, the taller the call stack would be.
|
| | before react 17 this was working pretty good now is unstable 100%, I can't even work
|
| I don't know what this means.
|
| It's very difficult for us to guess about what might be causing this issue without a repro. If you can't share your private app with us, can you make a reduced case that shows the problem on Code Sandbox?
what I mean with this
| before react 17 this was working pretty good now is unstable 100%, I can't even work
is... in react 16.3.1 this issue does not exist
| yes but production works perfect, just in dev mode is super slow.
|
| you can access here bringerparcel.dev
That is a production build, so it doesn't really help us reproduce the problem.
| A little more context: Even if only one component in your tree has a mutation effect, React will recurse from the root down to that component.
|
| ```
| A
| B C
| D E
| F G
| ```
|
| Let's say G has a mutation effect, React would still recurse from A -| B -| D -| G to apply the effect. The deeper your tree is, the taller the call stack would be.
|
| | before react 17 this was working pretty good now is unstable 100%, I can't even work
|
| I don't know what this means.
|
| It's very difficult for us to guess about what might be causing this issue without a repro. If you can't share your private app with us, can you make a reduced case that shows the problem on Code Sandbox?
i understand what react is doing but this is normal to take like almost 3 seconds per click ? that's not normal.
i have no enough time for reproduce this in code sandbox, but how I said, the error is very simple, every click event is taking forever
| @joacub Brian is saying it is absolutely _expected_ to see that function in the stack trace a lot — basically, once per `|div|` or a component that you're mounting. What would not be expected is seeing it more times than the nesting of your tree. It's hard to say anything from your screenshots — it would really help if you could either isolate the problematic behavior in an example or share your project or deploy it somewhere.
I understand, but even If I using React.memo in my component ? I guess yes, but im sure I don't have that nest tree at all, that's insane, I can't even reach the last call function in the tree, is endless.
| i understand what react is doing but this is normal to take like almost 3 seconds per click ? that's not normal.
This discussion isn't really productive without any way for us to repro the problem. We're kind of just going around in circles.
We have been using what will be version 17 of React at Facebook for several weeks (it changes a little each week) and we have not seen the behavior you're describing. So I would agree that it's not normal.
| | i understand what react is doing but this is normal to take like almost 3 seconds per click ? that's not normal.
|
| This discussion isn't really productive without any way for us to repro the problem. We're kind of just going around in circles.
|
| We have been using what will be version 17 of React at Facebook for several weeks (it changes a little each week) and we have not seen the behavior you're describing. So I would agree that it's not normal.
I will try get some time for reproduce this in codesandbox
i have n time for share in code sandbox but I test in my environment and I removed all the components in the tree and just leave just one, the behavior is the same super slow every onClick event, took the same time always, this is not related to the mutation effect in the tree, this is related to the events handler or something, how I said and for be clear, I leave just some components and the behavior is the same endless recursively calls.
i did more tests in my project removing more and more dev tools, is just react who has the issue, there is something related to state changes or events handler, sorry if I not been much useful but I have no much time for it. I Appreciate your concern and work.
Since you’re saying this is DEV-only, this information so far is not helpful without a project we can run locally.
I understand you can’t extract it to a sandbox, but maybe you could put a reduced version on GH?
@gaearon i've sent you an email to schedule a moment to test it with ngrok to give you access to localhost
@gaearon if you are able to do the test via ngrok we can do now.
Awesome, thanks. I'm not available now but I'll respond later.
---
I think I actually have a guess about what's going on. I think we're calling `invokeGuardedCallback` a lot more times than before.
Looking at RC1, we had code like this:
```js
do {
{
invokeGuardedCallback(null,commitLayoutEffects, null, root, lanes);
if (hasCaughtError()) {
if (!(nextEffect !== null)) {
{
throw Error( "Should be working on an effect." );
}
}
var _error2 = clearCaughtError();
captureCommitPhaseError(nextEffect, nextEffect.return, _error2);
nextEffect = nextEffect.nextEffect;
}
}
} while (nextEffect !== null);
```
but the actual `commitLayoutEffects` implementation had its own loop, avoiding repeated `invokeGuardedCallback` calls:
```js
function commitLayoutEffects(root, committedLanes) {
while (nextEffect !== null) {
setCurrentFiber(nextEffect);
var effectTag = nextEffect.effectTag;
if (effectTag | (Update | Callback)) {
var current = nextEffect.alternate;
commitLifeCycles(root, current, nextEffect);
}
{
if (effectTag | Ref) {
commitAttachRef(nextEffect);
}
}
resetCurrentFiber();
nextEffect = nextEffect.nextEffect;
}
}
```
Whereas in RC2, we recursively call `commitLayoutEffects` which wraps *every* effect into its own `invokeGuardedCallback`:
```js
function commitLayoutEffects(firstChild, root, committedLanes) {
var fiber = firstChild;
while (fiber !== null) {
if (fiber.child !== null) {
var primarySubtreeFlags = fiber.subtreeFlags | LayoutMask;
if (primarySubtreeFlags !== NoFlags) {
commitLayoutEffects(fiber.child, root, committedLanes);
}
}
{
setCurrentFiber(fiber);
invokeGuardedCallback(null, commitLayoutEffectsImpl, null, fiber, root, committedLanes);
if (hasCaughtError()) {
var error = clearCaughtError();
captureCommitPhaseError(fiber, fiber.return, error);
}
resetCurrentFiber();
}
fiber = fiber.sibling;
}
}
```
I can easily imagine this adding a lot of DEV overhead since it happens for every Fiber including host ones.
On WWW, `invokeGuardedCallback` is forked to go through a fancy try-catch, which is why we did not see it.
Seems like an easy fix. I'll look tomorrow.
@gaearon legend! is that
@gaearon 🥳🥳
| @gaearon legend! is that
can you please get in touch with me ? contact me at johan@webmediaprojects.com
Seems like we (React team) need to verify if the guarded callback is the source of slowness. We should be able to do that by disabling our internal "fancy try-catch" :smile: We'll keep thisthread posted.
| Seems like we (React team) need to verify if the guarded callback is the source of slowness. We should be able to do that by disabling our internal "fancy try-catch" 😄 We'll keep this thread posted.
do you have any ideas or we can change something really quick for at least mitigate for now this ?, _I did try somethings but no successfully._
To "mitigate" it? I believe this is only happening in DEV mode and only if you're using an RC, soif it's blocking your local development, don't use the RC until we've released another one.
@bvaughn well i have the same issue, i had to back to 16.13.1 and works perfectly is only since 17
@bvaughn in my case I can not go back, there is too many change in my app in more than 400 files, which is will cause me delays so, if you can suggest something for hot FIX this in the meantime I wait for the next RC I will appreciate.
Thanks
@joacub try to change to 16.13.1 in your package.json, RC versions supose to not be used in production, that one works perfectly i tested it
| @joacub try to change to 16.13.1 in your package.json, RC versions supose to not be used in production, that one works perfectly i tested it
I know, there is no option for me to go back now, that will be a waste or time for me. 😢
I don't understand how you have 400 files changed in a way that wouldn't work with v16. v17 doesn't add new APIs or behaviors. Why can't you just pin to a different version of React?
| I don't understand how you have 400 files changed in a way that wouldn't work with v16. v17 doesn't add new APIs or behaviors. Why can't you just pin to a different version of React?
The new jsx runtime,i have no anymore in my scope react on these file so i have to do a massive change, i can do with the IDE but i don’t think is going to save me time now, how i said that will be a waste of time
Just step back down to RC1 then. We'll release a new RC soon and you can try it again.
| Just step back down to RC1 then. We'll release a new RC soon and you can try it again.
I did, is slowly too 😭
| Just step back down to RC1then. We'll release a new RC soon and you can try it again.
just fyi, rc.1 is just a littler better, not much, still been slowly but not than rc.2
If you have no way to revert, you should not be upgrading to unstable versions. You should use source control and revert undesirable changes, or better yet, not merge them into master in the first place. I'm going to temporarily lock this thread because it's getting very noisy and we're not getting valuable information from it. We'll comment when we figure out the issue.
We've been able to reproduce the problem, so thanks for the report.
For now we've cut RC3 which doesn't have this issue, so please upgrade to get the fix.
In general though **you should never place yourself in a situation where you "need" a bugfix for an unstable release**. If you were not prepared to roll back to 16, you should not have run the codemodor upgraded to 17 RC. The whole point of RCs is that they may contain bugs, so they're not suitable for production deployments. And if you decide to ship it to users despite the risk, as soon as there's an issue you need to be prepared to roll back using version control.
| We've been able to reproduce the problem, so thanks for the report.
|
| For now we've cut RC3 which doesn't have this issue, so please upgrade to get the fix.
|
| In general though **you should never place yourself in a situation where you"need" a bugfix for an unstable release**. If you were not prepared to roll back to 16, you should not have run the codemod or upgraded to 17 RC. The whole point of RCs is that they may contain bugs, so they're not suitable for production deployments. And if you decide to ship it to users despite the risk, as soon as there's an issue you need to be prepared to roll back using version control.
Thanks, yes there is no problem, i go back to 17 rc 1, we are working with the last version because we want to release with the last version, thats does not block us at all, yeswe run codemon and thats was the only mistake, but even with that we still wanting this, we are in dev stage so we can do and is what we want. Thanks for your work , I appreciate
Can you verify whether RC3 also resolves the issue for you?
| Can you verify whether RC3 also resolves the issue for you?
Yes the issue was resolved, is working great now, thank you so much
@gaearon thank you i will take a look too
True class: duplicate
Text with highlighted words
Cursor initially appears in incorrect location on first focus in editor.
Challenge [Use an ID Attribute to Style an Element](https://www.freecodecamp.com/challenges/use-an-id-attribute-to-style-an-element#?solution=%3Clink%20href%3D%22https%3A%2F%2Ffonts.googleapis.com%2Fcss%3Ffamily%3DLobster%22%20rel%3D%22stylesheet%22%20type%3D%22text%2Fcss%22%3E%0A%3Cstyle%3E%0A%20%20.red-text%20%7B%0A%20%20%20%20color%3A%20red%3B%0A%20%20%7D%0A%20%20%23cat-photo-form%7B%0A%20%20%20%20background-color%3Agreen%3B%0A%20%20%7D%0A%20%20h2%20%7B%0A%20%20%20%20font-family%3A%20Lobster%2C%20Monospace%3B%0A%20%20%7D%0A%0A%20%20p%20%7B%0A%20%20%20%20font-size%3A%2016px%3B%0A%20%20%20%20font-family%3A%20Monospace%3B%0A%20%20%7D%0A%0A%20%20.thick-green-border%20%7B%0A%20%20%20%20border-color%3A%20green%3B%0A%20%20%20%20border-width%3A%2010px%3B%0A%20%20%20%20border-style%3A%20solid%3B%0A%20%20%20%20border-radius%3A%2050%25%3B%0A%20%20%7D%0A%0A%20%20.smaller-image%20%7B%0A%20%20%20%20width%3A%20100px%3B%0A%20%20%7D%0A%0A%20%20.gray-background%20%7B%0A%20%20%20%20background-color%3A%20gray%3B%0A%20%20%7D%0A%3C%2Fstyle%3E%0A%0A%3Ch2%20class%3D%22red-text%22%3ECatPhotoApp%3C%2Fh2%3E%0A%0A%3Cp%3EClick%20here%20for%20%3Ca%20href%3D%22%23%22%3Ecat%20photos%3C%2Fa%3E.%3C%2Fp%3E%0A%0A%3Ca%20href%3D%22%23%22%3E%3Cimg%20class%3D%22smaller-image%20thick-green-border%22%20alt%3D%22A%20cute%20orange%20cat%20lying%20on%20its%20back%22%20src%3D%22https%3A%2F%2Fbit.ly%2Ffcc-relaxing-cat%22%3E%3C%2Fa%3E%0A%0A%3Cdiv%20class%3D%22gray-background%22%3E%0A%20%20%3Cp%3EThings%20cats%20love%3A%3C%2Fp%3E%0A%20%20%3Cul%3E%0A%20%20%20%20%3Cli%3Ecat%20nip%3C%2Fli%3E%0A%20%20%20%20%3Cli%3Elaser%20pointers%3C%2Fli%3E%0A%20%20%20%20%3Cli%3Elasagna%3C%2Fli%3E%0A%20%20%3C%2Ful%3E%0A%20%20%3Cp%3ETop%203%20things%20cats%20hate%3A%3C%2Fp%3E%0A%20%20%3Col%3E%0A%20%20%20%20%3Cli%3Eflea%20treatment%3C%2Fli%3E%0A%20%20%20%20%3Cli%3Ethunder%3C%2Fli%3E%0A%20%20%20%20%3Cli%3Eother%20cats%3C%2Fli%3E%0A%20%20%3C%2Fol%3E%0A%3C%2Fdiv%3E%0A%0A%3Cform%20fccfaa%3D%22%2Fsubmit-cat-photo%22%20id%3D%22cat-photo-form%22%3E%0A%20%20%3Clabel%3E%3Cinput%20type%3D%22radio%22%20name%3D%22indoor-outdoor%22%20checked%3E%20Indoor%3C%2Flabel%3E%0A%20%20%3Clabel%3E%3Cinput%20type%3D%22radio%22%20name%3D%22indoor-outdoor%22%3E%20Outdoor%3C%2Flabel%3E%0A%20%20%3Clabel%3E%3Cinput%20type%3D%22checkbox%22%20name%3D%22personality%22%20checked%3E%20Loving%3C%2Flabel%3E%0A%20%20%3Clabel%3E%3Cinput%20type%3D%22checkbox%22%20name%3D%22personality%22%3E%20Lazy%3C%2Flabel%3E%0A%20%20%3Clabel%3E%3Cinput%20type%3D%22checkbox%22%20name%3D%22personality%22%3E%20Energetic%3C%2Flabel%3E%0A%20%20%3Cinput%20type%3D%22text%22%20placeholder%3D%22cat%20photo%20URL%22%20required%3E%0A%20%20%3Cbutton%20type%3D%22submit%22%3ESubmit%3C%2Fbutton%3E%0A%3C%2Fform%3E%0A) has an issue.
User Agent is: |code|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36|/code|.
Please describe how to reproduce this issue, and include links to screenshots if possible.
My code:
``` html
|link href="https://fonts.googleapis.com/css?family=Lobster" rel="stylesheet" type="text/css"|
|style|
.red-text {
color: red;
}
#cat-photo-form{
background-color:green;
}
h2 {
font-family: Lobster, Monospace;
}
p {
font-size: 16px;
font-family: Monospace;
}
.thick-green-border {
border-color: green;
border-width: 10px;
border-style: solid;
border-radius: 50%;
}
.smaller-image {
width: 100px;
}
.gray-background {
background-color: gray;
}
|/style|
|h2 class="red-text"|CatPhotoApp|/h2|
|p|Click here for |a href="#"|cat photos|/a|.|/p|
|a href="#"||img class="smaller-image thick-green-border" alt="A cute orange cat lying on its back" src="https://bit.ly/fcc-relaxing-cat"||/a|
|div class="gray-background"|
|p|Things cats love:|/p|
|ul|
|li|cat nip|/li|
|li|laser pointers|/li|
|li|lasagna|/li|
|/ul|
|p|Top 3 things cats hate:|/p|
|ol|
|li|flea treatment|/li|
|li|thunder|/li|
|li|other cats|/li|
|/ol|
|/div|
|form action="/submit-cat-photo" id="cat-photo-form"|
|label||input type="radio" name="indoor-outdoor" checked| Indoor|/label|
|label||input type="radio" name="indoor-outdoor"| Outdoor|/label|
|label||input type="checkbox" name="personality" checked| Loving|/label||label||input type="checkbox" name="personality"| Lazy|/label|
|label||input type="checkbox" name="personality"| Energetic|/label|
|input type="text" placeholder="cat photo URL" required|
|button type="submit"|Submit|/button|
|/form|
```
@miguels4ntos Please name your issue appropriately.
A better title might be "Cursor initially appears in incorrect location on first focus in editor"
This happens for me, too, on various challenges, only when clicking one of the last lines in the editor. Chrome/Mac
@BKinahan Done! Thanks!
空Cursor jumps randomly in Code Editor.
In all the exercises, we the users are forced to press the enter key before writing any code.
----
#### Update:
We have locked the conversation temporarily on this thread to collaborators only, this has been resolved in staging, and will be live soon.
The fix can be confirmed on the beta website.
The workaround currently on production website is:
Press the |kbd|Enter|/kbd| key on the challenge editor and then proceed with the challenge.
Apologies for the inconvenience meanwhile.
Reach us in the chat room if you need any assistance.
I could not reproduce this.
Could you provide some more information about your device / browser? I'm able to type within the editor as soon as a challenge loads, with the editor automatically taking focus, without needing to press enter.
It could be an issue specific to your setup.
I think I'm having the same issue with the editor.
In this exercise (please see image below) for example, I'm not able to focus some parts of the code before hitting enter (Neither with the mouse, nor with the keyboard). No matter where I'm clicking below line 54, the cursor jumps right to the beginning of line 54. After hitting "enter" once anywhere in the code, it all works fine!
I'm running latest Version of Chrome on Windows 10. I've noticed this issue in most of the exercices so far.
I have the same issue as smlabt no matter what exercise i'm doing. I'm using Firefox on Windows 10.
Line 56 input|would that work
For anyone having this issue please post your browser, os and challenge you first experienced this on. If possible a video of a GIF of the problem would be extremely helpful. Without this information the issue cannot be fixed.
Like I said I'm using Chrome on Windows 10.
Please try to reproduce the issue with a smaller window. I can imagine that this behaviour could have been caused by the line breaks.
I will try to create a gif asap.
@smlabt don't worry about the gif i have successfully reproduced it. Thank you for the help!
In some tasks the cursor won't go till last line ,it stopped going down before 3-5 line above ,so i had to make space above to complete it .(occurred mostly in html-css tasks)
My user agent is Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 @BKinahan
Using this for the first time and this usability issue it the first thing I noticed. It's true for all challenges I've tried so far in the javascript path. The _only_ way I can get to the code window is to click in. This is also true for scrolling the instructions panel.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36
I just started learning to code on Code Free Camp. As I literally just started I am a bit anxious in even trying to offer my help in diagnosing the issue.
What I can offer is my experience. Each time I load a page, I click onto the code window and try to type at the clicked location.
The actual location that it types at is bellow (seeming randomly) a short distance down from where I originally clicked.
The only time the cursor failed to jump down was when I clicked on first line in the editor. The one it starts at when the page loads.
(edit)
I also wanted to add. The problem seemed to not be an issue again starting on "https://www.freecodecamp.com/challenges/create-a-bootstrap-headline"
but its still an issue if I go back to the previous lesson.
True class: left
Text with highlighted words
Pushs and rebuilds fail after server was running out of disk space.
Our server was running out of disk space due to some docker diff files piling up (it seems they need to be cleaned out regularly, but that is a different topic). After making some space, `ps:rebuild` is not working anymore, it just returns immediately with no output. And we can also not push anymore to the server, we get:
```
! [remote rejected] dokku -| master (pre-receive hook declined)
error: failed to push some refs to 'dokku@.....:.....'
```
Help is greatly appreciated.
dokku report
-----| uname: Linux ubuntu-512mb-fra1-01 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
-----| memory:
total used free shared buffers cached
Mem: 994 647 346 10 19 111
-/+ buffers/cache: 515 478
Swap: 999 112 887
-----| docker version:
Client:
Version: 17.05.0-ce
API version: 1.29
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:06:06 2017
OS/Arch: linux/amd64
Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:06:06 2017
OS/Arch: linux/amd64
Experimental: false
-----| docker daemon info:
Containers: 12
Running: 5
Paused: 0
Stopped: 7
Images: 32
Server Version: 17.05.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 1133
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
apparmor
Kernel Version: 3.13.0-77-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 994MiB
Name: ubuntu-512mb-fra1-01
ID: 5XQF:UC63:3ZUV:GI2Y:JEWP:NJHA:NQ6X:5IKC:RRFA:XLTA:IILX:HBUZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): true
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
-----| sigil version: 0.4.0
-----| herokuish version:
herokuish: 0.3.31
buildpacks:
heroku-buildpack-multi v1.0.0
heroku-buildpack-ruby v163
heroku-buildpack-nodejs v99
heroku-buildpack-clojure v76
heroku-buildpack-python v109
heroku-buildpack-java v52
heroku-buildpack-gradle v22
heroku-buildpack-grails v21
heroku-buildpack-scala v76
heroku-buildpack-play v26
heroku-buildpack-php v121
heroku-buildpack-go v69
heroku-buildpack-erlang fa17af9
buildpack-nginx v8
-----| dokku version: 0.5.7
-----| dokku plugins:
plugn: 0.3.0
00_dokku-standard 0.5.7 enabled dokku core standard plugin
20_events 0.5.7 enabled dokku core events logging plugin
apps 0.5.7 enabled dokku core apps plugin
build-env 0.5.7 enabled dokku core build-env plugin
certs 0.5.7 enabled dokku core certificate management plugin
checks 0.5.7 enabled dokku core checks plugin
common 0.5.7 enabled dokku core common plugin
config 0.5.7 enabled dokku core config plugin
docker-options 0.5.7 enabled dokku core docker-options plugin
domains 0.5.7 enabled dokku core domains plugin
enter 0.5.7 enabled dokku core enter plugin
git 0.5.7 enabled dokku core git plugin
letsencrypt 0.8.6 enabled Automated installation of let's encrypt TLS certificates
logs 0.5.7 enabled dokku core logs plugin
memcached 1.0.0 enabled dokku memcached service plugin
named-containers 0.5.7 enabled dokku core named containers plugin
nginx-vhosts 0.5.7 enabled dokku core nginx-vhosts plugin
plugin 0.5.7 enabled dokku core plugin plugin
postgres 1.0.0 enabled dokku postgres service plugin
proxy 0.5.7 enabled dokku core proxy plugin
ps 0.5.7 enabled dokku core ps plugin
shell 0.5.7 enabled dokku core shell plugin
storage 0.5.7 enabled dokku core storage plugin
tags 0.5.7 enabled dokku core tags plugin
tar 0.5.7 enabled dokku core tar plugin
Environment details (AWS, VirtualBox, physical, etc.): Digital Ocean
Can you turn on trace mode and try deploying?
Okay, this is what I get when pushing:
```
+ case "$(lsb_release -si)" in
++ lsb_release -si
+ export DOKKU_DISTRO=ubuntu
+ DOKKU_DISTRO=ubuntu
+ export DOKKU_IMAGE=gliderlabs/herokuish
+ DOKKU_IMAGE=gliderlabs/herokuish
+ export DOKKU_LIB_ROOT=/var/lib/dokku
+ DOKKU_LIB_ROOT=/var/lib/dokku
+ export PLUGIN_PATH=/var/lib/dokku/plugins
+ PLUGIN_PATH=/var/lib/dokku/plugins
+ export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ export DOKKU_API_VERSION=1
+ DOKKU_API_VERSION=1
+ export DOKKU_NOT_IMPLEMENTED_EXIT=10
+ DOKKU_NOT_IMPLEMENTED_EXIT=10
+ export DOKKU_VALID_EXIT=0
+ DOKKU_VALID_EXIT=0
+ export DOKKU_LOGS_DIR=/var/log/dokku
+ DOKKU_LOGS_DIR=/var/log/dokku
+ export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ export DOKKU_CONTAINER_LABEL=dokku
+ DOKKU_CONTAINER_LABEL=dokku
+ export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ parse_args git-receive-pack ''\''myapp'\'''
+ declare 'desc=top-level cli arg parser'
+ local next_index=1
+ local skip=false
+ args=("$@")
+ local args
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=2
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=3
+ return 0
+ args=("$@")
+ [[ git-receive-pack =~ ^--.* ]]
+ has_tty
+ declare 'desc=return 0 if we have a tty'
++ /usr/bin/tty
++ true
+ [[ not a tty == \n\o\t\ \a\ \t\t\y ]]
+ return 1
+ DOKKU_QUIET_OUTPUT=1
++ id -un
+ [[ dokku != \d\o\k\k\u ]]
++ id -un
+ [[ dokku != \r\o\o\t ]]
+ [[ git-receive-pack =~ ^plugin:.* ]]
+ [[ -n git-receive-pack 'myapp' ]]
+ export -n SSH_ORIGINAL_COMMAND
+ [[ git-receive-pack =~ config-* ]]
+ [[ git-receive-pack =~ docker-options* ]]
+ set -f
+ /usr/local/bin/dokku git-receive-pack ''\''myapp'\'''
+ case "$(lsb_release -si)" in
++ lsb_release -si
+ export DOKKU_DISTRO=ubuntu
+ DOKKU_DISTRO=ubuntu
+ export DOKKU_IMAGE=gliderlabs/herokuish
+ DOKKU_IMAGE=gliderlabs/herokuish
+ export DOKKU_LIB_ROOT=/var/lib/dokku
+ DOKKU_LIB_ROOT=/var/lib/dokku
+ export PLUGIN_PATH=/var/lib/dokku/plugins
+ PLUGIN_PATH=/var/lib/dokku/plugins
+ export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ export DOKKU_API_VERSION=1
+ DOKKU_API_VERSION=1
+ export DOKKU_NOT_IMPLEMENTED_EXIT=10
+ DOKKU_NOT_IMPLEMENTED_EXIT=10
+ export DOKKU_VALID_EXIT=0
+ DOKKU_VALID_EXIT=0
+ export DOKKU_LOGS_DIR=/var/log/dokku
+ DOKKU_LOGS_DIR=/var/log/dokku
+ export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ export DOKKU_CONTAINER_LABEL=dokku
+ DOKKU_CONTAINER_LABEL=dokku
+ export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ parse_args git-receive-pack ''\''myapp'\'''
+ declare 'desc=top-level cli arg parser'
+ local next_index=1
+ local skip=false
+ args=("$@")
+ local args
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=2
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=3
+ return 0
+ args=("$@")
+ [[ git-receive-pack =~ ^--.* ]]
+ has_tty
+ declare 'desc=return 0 if we have a tty'
++ /usr/bin/tty
++ true
+ [[ not a tty == \n\o\t\ \a\ \t\t\y ]]
+ return 1
+ DOKKU_QUIET_OUTPUT=1
++ id -un
+ [[ dokku != \d\o\k\k\u ]]
++ id -un
+ [[ dokku != \r\o\o\t ]]
+ [[ git-receive-pack =~ ^plugin:.* ]]
+ [[ -n '' ]]
+ dokku_auth git-receive-pack ''\''myapp'\'''
+ declare 'desc=calls user-auth plugin trigger'
+ export SSH_USER=dokku
+ SSH_USER=dokku
+ export 'SSH_NAME=[nico]'
+ SSH_NAME='[nico]'
+ plugn trigger user-auth dokku '[nico]' git-receive-pack ''\''myapp'\'''
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ [[ ! -n '' ]]
+ return 0
+ case "$1" in
+ execute_dokku_cmd git-receive-pack ''\''myapp'\'''
+ declare 'desc=executes dokku sub-commands'
+ local PLUGIN_NAME=git-receive-pack
+ local PLUGIN_CMD=git-receive-pack
+ local implemented=0
+ local script
+ argv=("$@")
+ local argv
+ case "$PLUGIN_NAME" in
++ readlink -f /var/lib/dokku/plugins/enabled/git-receive-pack
+ [[ /var/lib/dokku/plugins/enabled/git-receive-pack == *core-plugins* ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-receive-pack/subcommands/default ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-receive-pack/subcommands/git-receive-pack ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-receive-pack/subcommands/git-receive-pack ]]
+ [[ 0 -eq 0 ]]
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/00_dokku-standard/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/20_events/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/apps/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/certs/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/checks/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/config/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/docker-options/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/domains/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/enter/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/git/commands git-receive-pack ''\''myapp'\'''
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ source /var/lib/dokku/plugins/available/apps/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++ source /var/lib/dokku/core-plugins/available/common/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
++ source /var/lib/dokku/plugins/available/config/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
+++ source /var/lib/dokku/core-plugins/available/common/functions
++++ set -eo pipefail
++++ [[ -n 1 ]]
++++ set -x
+ source /var/lib/dokku/plugins/available/config/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++ source /var/lib/dokku/core-plugins/available/common/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
+ case "$1" in
+ git_glob_cmd git-receive-pack ''\''myapp'\'''
+ declare 'desc=catch-all for any other git-* commands'
+ local 'cmd=git-*'
++ sed 's/^\///g'
++ sed 's/\\'\''/'\''/g'
++ perl -pe 's/(?|!\\)'\''//g'
++ echo ''\''myapp'\'''
+ local APP=myapp
+ local APP_PATH=/home/dokku/myapp
+ [[ git-receive-pack == \g\i\t\-\r\e\c\e\i\v\e\-\p\a\c\k ]]
+ [[ ! -d /home/dokku/myapp/refs ]]
+ [[ git-receive-pack == \g\i\t\-\r\e\c\e\i\v\e\-\p\a\c\k ]]
+ local 'args=git-receive-pack '\''/home/dokku/myapp'\'''
+ git-shell -c 'git-receive-pack '\''/home/dokku/myapp'\'''
Total 0 (delta 0), reused 0 (delta 0)
remote: + case "$(lsb_release -si)" in
remote: ++ lsb_release -si
remote: + export DOKKU_DISTRO=ubuntu
remote: + DOKKU_DISTRO=ubuntu
remote: + export DOKKU_IMAGE=gliderlabs/herokuish
remote: + DOKKU_IMAGE=gliderlabs/herokuish
remote: + export DOKKU_LIB_ROOT=/var/lib/dokku
remote: + DOKKU_LIB_ROOT=/var/lib/dokku
remote: + export PLUGIN_PATH=/var/lib/dokku/plugins
remote: + PLUGIN_PATH=/var/lib/dokku/plugins
remote: + export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
remote: + PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
remote: + export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
remote: + PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
remote: + export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
remote: + PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
remote: + export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
remote: + PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
remote: + export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
remote: + PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
remote: + export DOKKU_API_VERSION=1
remote: + DOKKU_API_VERSION=1
remote: + export DOKKU_NOT_IMPLEMENTED_EXIT=10
remote: + DOKKU_NOT_IMPLEMENTED_EXIT=10
remote: + export DOKKU_VALID_EXIT=0
remote: + DOKKU_VALID_EXIT=0
remote: + export DOKKU_LOGS_DIR=/var/log/dokku
remote: + DOKKU_LOGS_DIR=/var/log/dokku
remote: + export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
remote: + DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
remote: + export DOKKU_CONTAINER_LABEL=dokku
remote: + DOKKU_CONTAINER_LABEL=dokku
remote: + export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
remote: + DOKKU_GLOBAL_RUN_ARGS=--label=dokku
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + parse_args git-hook myapp
remote: + declare 'desc=top-level cli arg parser'
remote: + local next_index=1
remote: + local skip=false
remote: + args=("$@")
remote: + local args
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=2
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=3
remote: + return 0
remote: + args=("$@")
remote: + [[ git-hook =~ ^--.* ]]
remote: + has_tty
remote: + declare 'desc=return 0 if we have a tty'
remote: ++ /usr/bin/tty
remote: ++ true
remote: + [[ not a tty == \n\o\t\ \a\ \t\t\y ]]
remote: + return 1
remote: + DOKKU_QUIET_OUTPUT=1
remote: ++ id -un
remote: + [[ dokku != \d\o\k\k\u ]]
remote: ++ id -un
remote: + [[ dokku != \r\o\o\t ]]
remote: + [[ git-hook =~ ^plugin:.* ]]
remote: + [[ -n '' ]]
remote: + dokku_auth git-hook myapp
remote: + declare 'desc=calls user-auth plugin trigger'
remote: + export SSH_USER=dokku
remote: + SSH_USER=dokku
remote: + export 'SSH_NAME=[nico]'
remote: + SSH_NAME='[nico]'
remote: + plugn trigger user-auth dokku '[nico]' git-hook myapp
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + [[ ! -n '' ]]
remote: + return 0
remote: + case "$1" in
remote: + execute_dokku_cmd git-hook myapp
remote: + declare 'desc=executes dokku sub-commands'
remote: + local PLUGIN_NAME=git-hook
remote: + local PLUGIN_CMD=git-hook
remote: + local implemented=0
remote: + local script
remote: + argv=("$@")
remote: + local argv
remote: + case "$PLUGIN_NAME" in
remote: ++ readlink -f /var/lib/dokku/plugins/enabled/git-hook
remote: + [[ /var/lib/dokku/plugins/enabled/git-hook == *core-plugins* ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-hook/subcommands/default ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-hook/subcommands/git-hook ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-hook/subcommands/git-hook ]]
remote: + [[ 0 -eq 0 ]]
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/00_dokku-standard/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/20_events/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/apps/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/certs/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/checks/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/config/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/docker-options/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/domains/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/enter/commands git-hook myapp
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/git/commands git-hook myapp
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + source /var/lib/dokku/plugins/available/apps/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: ++ source /var/lib/dokku/core-plugins/available/common/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: ++ source /var/lib/dokku/plugins/available/config/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: +++ source /var/lib/dokku/core-plugins/available/common/functions
remote: ++++ set -eo pipefail
remote: ++++ [[ -n 1 ]]
remote: ++++ set -x
remote: + source /var/lib/dokku/plugins/available/config/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: ++ source /var/lib/dokku/core-plugins/available/common/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: + case "$1" in
remote: + git_hook_cmd git-hook myapp
remote: + declare 'desc=kick off receive-app trigger from git prereceive hook'
remote: + local cmd=git-hook
remote: + local APP=myapp
remote: + local oldrev newrev refname
remote: + read -r oldrev newrev refname
remote: + [[ refs/heads/master = \r\e\f\s\/\h\e\a\d\s\/\m\a\s\t\e\r ]]
remote: + plugn trigger receive-app myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + [[ ! -n '' ]]
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + git_receive_app myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=git receive-app plugin trigger'
remote: + local trigger=git_receive_app
remote: + local APP=myapp
remote: + local REV=b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + [[ ! -d /home/dokku/myapp/refs ]]
remote: + dokku git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + case "$(lsb_release -si)" in
remote: ++ lsb_release -si
remote: + export DOKKU_DISTRO=ubuntu
remote: + DOKKU_DISTRO=ubuntu
remote: + export DOKKU_IMAGE=gliderlabs/herokuish
remote: + DOKKU_IMAGE=gliderlabs/herokuish
remote: + export DOKKU_LIB_ROOT=/var/lib/dokku
remote: + DOKKU_LIB_ROOT=/var/lib/dokku
remote: + export PLUGIN_PATH=/var/lib/dokku/plugins
remote: + PLUGIN_PATH=/var/lib/dokku/plugins
remote: + export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
remote: + PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
remote: + export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
remote: + PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
remote: + export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
remote: + PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
remote: + export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
remote: + PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
remote: + export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
remote: + PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
remote: + export DOKKU_API_VERSION=1
remote: + DOKKU_API_VERSION=1
remote: + export DOKKU_NOT_IMPLEMENTED_EXIT=10
remote: + DOKKU_NOT_IMPLEMENTED_EXIT=10
remote: + export DOKKU_VALID_EXIT=0
remote: + DOKKU_VALID_EXIT=0
remote: + export DOKKU_LOGS_DIR=/var/log/dokku
remote: + DOKKU_LOGS_DIR=/var/log/dokku
remote: + export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
remote: + DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
remote: + export DOKKU_CONTAINER_LABEL=dokku
remote: + DOKKU_CONTAINER_LABEL=dokku
remote: + export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
remote: + DOKKU_GLOBAL_RUN_ARGS=--label=dokku
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + parse_args git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=top-level cli arg parser'
remote: + local next_index=1
remote: + local skip=false
remote: + args=("$@")
remote: + local args
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=2
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=3
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=4
remote: + return 0
remote: + args=("$@")
remote: + [[ git-build =~ ^--.* ]]
remote: + has_tty
remote: + declare 'desc=return 0 if we have a tty'
remote: ++ /usr/bin/tty
remote: ++ true
remote: + [[ not a tty == \n\o\t\ \a\ \t\t\y ]]
remote: + return 1
remote: + DOKKU_QUIET_OUTPUT=1
remote: ++ id -un
remote: + [[ dokku != \d\o\k\k\u ]]
remote: ++ id -un
remote: + [[ dokku != \r\o\o\t ]]
remote: + [[ git-build =~ ^plugin:.* ]]
remote: + [[ -n '' ]]
remote: + dokku_auth git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=calls user-auth plugin trigger'
remote: + export SSH_USER=dokku
remote: + SSH_USER=dokku
remote: + export 'SSH_NAME=[nico]'
remote: + SSH_NAME='[nico]'
remote: + plugn trigger user-auth dokku '[nico]' git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + [[ ! -n '' ]]
remote: + return 0
remote: + case "$1" in
remote: + execute_dokku_cmd git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=executes dokku sub-commands'
remote: + local PLUGIN_NAME=git-build
remote: + local PLUGIN_CMD=git-build
remote: + local implemented=0
remote: + local script
remote: + argv=("$@")
remote: + local argv
remote: + case "$PLUGIN_NAME" in
remote: ++ readlink -f /var/lib/dokku/plugins/enabled/git-build
remote: + [[ /var/lib/dokku/plugins/enabled/git-build == *core-plugins* ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-build/subcommands/default ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-build/subcommands/git-build ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-build/subcommands/git-build ]]
remote: + [[ 0 -eq 0 ]]
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/00_dokku-standard/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/20_events/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/apps/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/certs/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/checks/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/config/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/docker-options/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/domains/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/enter/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/git/commands git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + source /var/lib/dokku/plugins/available/apps/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: ++ source /var/lib/dokku/core-plugins/available/common/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: ++ source /var/lib/dokku/plugins/available/config/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: +++ source /var/lib/dokku/core-plugins/available/common/functions
remote: ++++ set -eo pipefail
remote: ++++ [[ -n 1 ]]
remote: ++++ set -x
remote: + source /var/lib/dokku/plugins/available/config/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: ++ source /var/lib/dokku/core-plugins/available/common/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: + case "$1" in
remote: + git_build_cmd git-build myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=lock git-build'
remote: + local cmd=git-build
remote: + local APP=myapp
remote: + local APP_BUILD_LOCK=/home/dokku/myapp/.build.lock
remote: + local 'APP_BUILD_LOCK_MSG=myapp is currently being deployed or locked. Waiting...'
remote: ++ flock -n /home/dokku/myapp/.build.lock true
remote: ++ echo 0
remote: + [[ 0 -ne 0 ]]
remote: + shift 1
remote: + flock -o /home/dokku/myapp/.build.lock dokku git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + case "$(lsb_release -si)" in
remote: ++ lsb_release -si
remote: + export DOKKU_DISTRO=ubuntu
remote: + DOKKU_DISTRO=ubuntu
remote: + export DOKKU_IMAGE=gliderlabs/herokuish
remote: + DOKKU_IMAGE=gliderlabs/herokuish
remote: + export DOKKU_LIB_ROOT=/var/lib/dokku
remote: + DOKKU_LIB_ROOT=/var/lib/dokku
remote: + export PLUGIN_PATH=/var/lib/dokku/plugins
remote: + PLUGIN_PATH=/var/lib/dokku/plugins
remote: + export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
remote: + PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
remote: + export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
remote: + PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
remote: + export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
remote: + PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
remote: + export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
remote: + PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
remote: + export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
remote: + PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
remote: + export DOKKU_API_VERSION=1
remote: + DOKKU_API_VERSION=1
remote: + export DOKKU_NOT_IMPLEMENTED_EXIT=10
remote: + DOKKU_NOT_IMPLEMENTED_EXIT=10
remote: + export DOKKU_VALID_EXIT=0
remote: + DOKKU_VALID_EXIT=0
remote: + export DOKKU_LOGS_DIR=/var/log/dokku
remote: + DOKKU_LOGS_DIR=/var/log/dokku
remote: + export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
remote: + DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
remote: + export DOKKU_CONTAINER_LABEL=dokku
remote: + DOKKU_CONTAINER_LABEL=dokku
remote: + export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
remote: + DOKKU_GLOBAL_RUN_ARGS=--label=dokku
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + parse_args git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=top-level cli arg parser'
remote: + local next_index=1
remote: + local skip=false
remote: + args=("$@")
remote: + local args
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=2
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=3
remote: + for arg in '"$@"'
remote: + false
remote: + case "$arg" in
remote: + local next_index=4
remote: + return 0
remote: + args=("$@")
remote: + [[ git-build-locked =~ ^--.* ]]
remote: + has_tty
remote: + declare 'desc=return 0 if we have a tty'
remote: ++ /usr/bin/tty
remote: ++ true
remote: + [[ not a tty == \n\o\t\ \a\ \t\t\y ]]
remote: + return 1
remote: + DOKKU_QUIET_OUTPUT=1
remote: ++ id -un
remote: + [[ dokku != \d\o\k\k\u ]]
remote: ++ id -un
remote: + [[ dokku != \r\o\o\t ]]
remote: + [[ git-build-locked =~ ^plugin:.* ]]
remote: + [[ -n '' ]]
remote: + dokku_auth git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=calls user-auth plugin trigger'
remote: + export SSH_USER=dokku
remote: + SSH_USER=dokku
remote: + export 'SSH_NAME=[nico]'
remote: + SSH_NAME='[nico]'
remote: + plugn trigger user-auth dokku '[nico]' git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + [[ ! -n '' ]]
remote: + return 0
remote: + case "$1" in
remote: + execute_dokku_cmd git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=executes dokku sub-commands'
remote: + local PLUGIN_NAME=git-build-locked
remote: + local PLUGIN_CMD=git-build-locked
remote: + local implemented=0
remote: + local script
remote: + argv=("$@")
remote: + local argv
remote: + case "$PLUGIN_NAME" in
remote: ++ readlink -f /var/lib/dokku/plugins/enabled/git-build-locked
remote: + [[ /var/lib/dokku/plugins/enabled/git-build-locked == *core-plugins* ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-build-locked/subcommands/default ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-build-locked/subcommands/git-build-locked ]]
remote: + [[ -x /var/lib/dokku/plugins/enabled/git-build-locked/subcommands/git-build-locked ]]
remote: + [[ 0 -eq 0 ]]
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/00_dokku-standard/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/20_events/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/apps/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/certs/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/checks/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/config/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/docker-options/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/domains/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/enter/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + exit_code=10
remote: + set -e
remote: + [[ 10 -eq 10 ]]
remote: + continue
remote: + for script in '$PLUGIN_ENABLED_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/enabled/git/commands git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + source /var/lib/dokku/core-plugins/available/common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + source /var/lib/dokku/plugins/available/apps/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: ++ source /var/lib/dokku/core-plugins/available/common/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: ++ source /var/lib/dokku/plugins/available/config/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: +++ source /var/lib/dokku/core-plugins/available/common/functions
remote: ++++ set -eo pipefail
remote: ++++ [[ -n 1 ]]
remote: ++++ set -x
remote: + source /var/lib/dokku/plugins/available/config/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: ++ source /var/lib/dokku/core-plugins/available/common/functions
remote: +++ set -eo pipefail
remote: +++ [[ -n 1 ]]
remote: +++ set -x
remote: + case "$1" in
remote: + git_build_locked_cmd git-build-locked myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=setup and call git_build_app_repo'
remote: + local cmd=git-build-locked
remote: + local APP=myapp
remote: + [[ 3 -ge 3 ]]
remote: + local REF=b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + git_build_app_repo myapp b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + declare 'desc=builds local git app repo for app'
remote: + verify_app_name myapp
remote: + declare 'desc=verify app name format and app existence'
remote: + local APP=myapp
remote: + [[ ! -n myapp ]]
remote: + [[ ! myapp =~ ^[a-z].* ]]
remote: + [[ ! -d /home/dokku/myapp ]]
remote: + return 0
remote: + local APP=myapp
remote: + local REV=b4287d41e948d5bd423edd1d5528d86784678d8c
remote: ++ mktemp -d /tmp/dokku_git.XXXX
remote: + local GIT_BUILD_APP_REPO_TMP_WORK_DIR=/tmp/dokku_git.kXDG
remote: + trap 'rm -rf "$GIT_BUILD_APP_REPO_TMP_WORK_DIR" | /dev/null' RETURN INT TERM EXIT
remote: + local TMP_TAG=dokku/b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + chmod 755 /tmp/dokku_git.kXDG
remote: + unset GIT_DIR GIT_WORK_TREE
remote: + pushd /tmp/dokku_git.kXDG
remote: + [[ ! -d /home/dokku/myapp ]]
remote: + GIT_DIR=/home/dokku/myapp
remote: + git tag -d dokku/b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + GIT_DIR=/home/dokku/myapp
remote: + git tag dokku/b4287d41e948d5bd423edd1d5528d86784678d8c b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + git init
remote: + git config advice.detachedHead false
remote: + git remote add origin /home/dokku/myapp
remote: + git fetch --depth=1 origin refs/tags/dokku/b4287d41e948d5bd423edd1d5528d86784678d8c
remote: + rm -rf /tmp/dokku_git.kXDG
remote: + exit_code=141
remote: + set -e
remote: + [[ 141 -eq 10 ]]
remote: + implemented=1
remote: + [[ 141 -ne 0 ]]
remote: + exit 141
remote: + exit_code=141
remote: + set -e
remote: + [[ 141 -eq 10 ]]
remote: + implemented=1
remote: + [[ 141 -ne 0 ]]
remote: + exit 141
remote: + exit_code=141
remote: + set -e
remote: + [[ 141 -eq 10 ]]
remote: + implemented=1
remote: + [[ 141 -ne 0 ]]
remote: + exit 141
+ exit_code=0
+ set -e
+ [[ 0 -eq 10 ]]
+ implemented=1
+ [[ 0 -ne 0 ]]
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/letsencrypt/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/logs/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/memcached/commands git-receive-pack ''\''myapp'\'''
+ PLUGIN_BASE_PATH=/var/lib/dokku/plugins
+ [[ -n 1 ]]
+ PLUGIN_BASE_PATH=/var/lib/dokku/plugins/enabled
+ source /var/lib/dokku/plugins/enabled/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++ dirname /var/lib/dokku/plugins/enabled/memcached/commands
+ source /var/lib/dokku/plugins/enabled/memcached/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+++ dirname /var/lib/dokku/plugins/enabled/memcached/commands
++ source /var/lib/dokku/plugins/enabled/memcached/config
+++ export MEMCACHED_IMAGE=memcached
+++ MEMCACHED_IMAGE=memcached
+++ export MEMCACHED_IMAGE_VERSION=1.4.25
+++ MEMCACHED_IMAGE_VERSION=1.4.25
+++ export MEMCACHED_ROOT=/var/lib/dokku/services/memcached
+++ MEMCACHED_ROOT=/var/lib/dokku/services/memcached
+++ export PLUGIN_COMMAND_PREFIX=memcached
+++ PLUGIN_COMMAND_PREFIX=memcached
+++ export PLUGIN_DATA_ROOT=/var/lib/dokku/services/memcached
+++ PLUGIN_DATA_ROOT=/var/lib/dokku/services/memcached
+++ PLUGIN_DATASTORE_PORTS=(11211)
+++ export PLUGIN_DATASTORE_PORTS
+++ export PLUGIN_DEFAULT_ALIAS=MEMCACHED
+++ PLUGIN_DEFAULT_ALIAS=MEMCACHED
+++ export PLUGIN_ALT_ALIAS=DOKKU_MEMCACHED
+++ PLUGIN_ALT_ALIAS=DOKKU_MEMCACHED
+++ export PLUGIN_IMAGE=memcached
+++ PLUGIN_IMAGE=memcached
+++ export PLUGIN_IMAGE_VERSION=1.4.25
+++ PLUGIN_IMAGE_VERSION=1.4.25
+++ export PLUGIN_SCHEME=memcached
+++ PLUGIN_SCHEME=memcached
+++ export PLUGIN_SERVICE=Memcached
+++ PLUGIN_SERVICE=Memcached
++ dirname /var/lib/dokku/plugins/enabled/memcached/commands
+ source /var/lib/dokku/plugins/enabled/memcached/config
++ export MEMCACHED_IMAGE=memcached
++ MEMCACHED_IMAGE=memcached
++ export MEMCACHED_IMAGE_VERSION=1.4.25
++ MEMCACHED_IMAGE_VERSION=1.4.25
++ export MEMCACHED_ROOT=/var/lib/dokku/services/memcached
++ MEMCACHED_ROOT=/var/lib/dokku/services/memcached
++ export PLUGIN_COMMAND_PREFIX=memcached
++ PLUGIN_COMMAND_PREFIX=memcached
++ export PLUGIN_DATA_ROOT=/var/lib/dokku/services/memcached
++ PLUGIN_DATA_ROOT=/var/lib/dokku/services/memcached
++ PLUGIN_DATASTORE_PORTS=(11211)
++ export PLUGIN_DATASTORE_PORTS
++ export PLUGIN_DEFAULT_ALIAS=MEMCACHED
++ PLUGIN_DEFAULT_ALIAS=MEMCACHED
++ export PLUGIN_ALT_ALIAS=DOKKU_MEMCACHED
++ PLUGIN_ALT_ALIAS=DOKKU_MEMCACHED
++ export PLUGIN_IMAGE=memcached
++ PLUGIN_IMAGE=memcached
++ export PLUGIN_IMAGE_VERSION=1.4.25
++ PLUGIN_IMAGE_VERSION=1.4.25
++ export PLUGIN_SCHEME=memcached
++ PLUGIN_SCHEME=memcached
++ export PLUGIN_SERVICE=Memcached
++ PLUGIN_SERVICE=Memcached
+ [[ git-receive-pack == memcached:* ]]
+ [[ -d /var/lib/dokku/services/memcached/* ]]
+ case "$1" in
+ exit 10
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/nginx-vhosts/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/plugin/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/postgres/commands git-receive-pack ''\''myapp'\'''
+ PLUGIN_BASE_PATH=/var/lib/dokku/plugins
+ [[ -n 1 ]]
+ PLUGIN_BASE_PATH=/var/lib/dokku/plugins/enabled
+ source /var/lib/dokku/plugins/enabled/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++ dirname /var/lib/dokku/plugins/enabled/postgres/commands
+ source /var/lib/dokku/plugins/enabled/postgres/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+++ dirname /var/lib/dokku/plugins/enabled/postgres/commands
++ source /var/lib/dokku/plugins/enabled/postgres/config
+++ export POSTGRES_IMAGE=postgres
+++ POSTGRES_IMAGE=postgres
+++ export POSTGRES_IMAGE_VERSION=9.5.0
+++ POSTGRES_IMAGE_VERSION=9.5.0
+++ export POSTGRES_ROOT=/var/lib/dokku/services/postgres
+++ POSTGRES_ROOT=/var/lib/dokku/services/postgres
+++ export PLUGIN_COMMAND_PREFIX=postgres
+++ PLUGIN_COMMAND_PREFIX=postgres
+++ export PLUGIN_DATA_ROOT=/var/lib/dokku/services/postgres
+++ PLUGIN_DATA_ROOT=/var/lib/dokku/services/postgres
+++ PLUGIN_DATASTORE_PORTS=(5432)
+++ export PLUGIN_DATASTORE_PORTS
+++ export PLUGIN_DEFAULT_ALIAS=DATABASE
+++ PLUGIN_DEFAULT_ALIAS=DATABASE
+++ export PLUGIN_ALT_ALIAS=DOKKU_POSTGRES
+++ PLUGIN_ALT_ALIAS=DOKKU_POSTGRES
+++ export PLUGIN_IMAGE=postgres
+++ PLUGIN_IMAGE=postgres
+++ export PLUGIN_IMAGE_VERSION=9.5.0
+++ PLUGIN_IMAGE_VERSION=9.5.0
+++ export PLUGIN_SCHEME=postgres
+++ PLUGIN_SCHEME=postgres
+++ export PLUGIN_SERVICE=Postgres
+++ PLUGIN_SERVICE=Postgres
++ dirname /var/lib/dokku/plugins/enabled/postgres/commands
+ source /var/lib/dokku/plugins/enabled/postgres/config
++ export POSTGRES_IMAGE=postgres
++ POSTGRES_IMAGE=postgres
++ export POSTGRES_IMAGE_VERSION=9.5.0
++ POSTGRES_IMAGE_VERSION=9.5.0
++ export POSTGRES_ROOT=/var/lib/dokku/services/postgres
++ POSTGRES_ROOT=/var/lib/dokku/services/postgres
++ export PLUGIN_COMMAND_PREFIX=postgres
++ PLUGIN_COMMAND_PREFIX=postgres
++ export PLUGIN_DATA_ROOT=/var/lib/dokku/services/postgres
++ PLUGIN_DATA_ROOT=/var/lib/dokku/services/postgres
++ PLUGIN_DATASTORE_PORTS=(5432)
++ export PLUGIN_DATASTORE_PORTS
++ export PLUGIN_DEFAULT_ALIAS=DATABASE
++ PLUGIN_DEFAULT_ALIAS=DATABASE
++ export PLUGIN_ALT_ALIAS=DOKKU_POSTGRES
++ PLUGIN_ALT_ALIAS=DOKKU_POSTGRES
++ export PLUGIN_IMAGE=postgres
++ PLUGIN_IMAGE=postgres
++ export PLUGIN_IMAGE_VERSION=9.5.0
++ PLUGIN_IMAGE_VERSION=9.5.0
++ export PLUGIN_SCHEME=postgres
++ PLUGIN_SCHEME=postgres
++ export PLUGIN_SERVICE=Postgres
++ PLUGIN_SERVICE=Postgres
+ [[ git-receive-pack == postgres:* ]]
+ [[ -d /var/lib/dokku/services/postgres/* ]]
+ case "$1" in
+ exit 10
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/proxy/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/ps/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/shell/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/storage/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/tags/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/tar/commands git-receive-pack ''\''myapp'\'''
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ [[ 1 -eq 0 ]]
+ set +f
+ exit 0
```
And this when rebuilding:
```
+ case "$(lsb_release -si)" in
++ lsb_release -si
+ export DOKKU_DISTRO=ubuntu
+ DOKKU_DISTRO=ubuntu
+ export DOKKU_IMAGE=gliderlabs/herokuish
+ DOKKU_IMAGE=gliderlabs/herokuish
+ export DOKKU_LIB_ROOT=/var/lib/dokku
+ DOKKU_LIB_ROOT=/var/lib/dokku
+ export PLUGIN_PATH=/var/lib/dokku/plugins
+ PLUGIN_PATH=/var/lib/dokku/plugins
+ export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ export DOKKU_API_VERSION=1
+ DOKKU_API_VERSION=1
+ export DOKKU_NOT_IMPLEMENTED_EXIT=10
+ DOKKU_NOT_IMPLEMENTED_EXIT=10
+ export DOKKU_VALID_EXIT=0
+ DOKKU_VALID_EXIT=0
+ export DOKKU_LOGS_DIR=/var/log/dokku
+ DOKKU_LOGS_DIR=/var/log/dokku
+ export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ export DOKKU_CONTAINER_LABEL=dokku
+ DOKKU_CONTAINER_LABEL=dokku
+ export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ parse_args ps:rebuild myapp
+ declare 'desc=top-level cli arg parser'
+ local next_index=1
+ local skip=false
+ args=("$@")
+ local args
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=2
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=3
+ return 0
+ args=("$@")
+ [[ ps:rebuild =~ ^--.* ]]
+ has_tty
+ declare 'desc=return 0 if we have a tty'
++ /usr/bin/tty
+ [[ /dev/pts/0 == \n\o\t\ \a\ \t\t\y ]]
+ return 0
++ id -un
+ [[ root != \d\o\k\k\u ]]
+ [[ ! ps:rebuild =~ plugin:* ]]
++ id -un
+ export SSH_USER=root
+ SSH_USER=root
+ sudo -u dokku -E -H /usr/local/bin/dokku ps:rebuild myapp
+ case "$(lsb_release -si)" in
++ lsb_release -si
+ export DOKKU_DISTRO=ubuntu
+ DOKKU_DISTRO=ubuntu
+ export DOKKU_IMAGE=gliderlabs/herokuish
+ DOKKU_IMAGE=gliderlabs/herokuish
+ export DOKKU_LIB_ROOT=/var/lib/dokku
+ DOKKU_LIB_ROOT=/var/lib/dokku
+ export PLUGIN_PATH=/var/lib/dokku/plugins
+ PLUGIN_PATH=/var/lib/dokku/plugins
+ export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ export DOKKU_API_VERSION=1
+ DOKKU_API_VERSION=1
+ export DOKKU_NOT_IMPLEMENTED_EXIT=10
+ DOKKU_NOT_IMPLEMENTED_EXIT=10
+ export DOKKU_VALID_EXIT=0
+ DOKKU_VALID_EXIT=0
+ export DOKKU_LOGS_DIR=/var/log/dokku
+ DOKKU_LOGS_DIR=/var/log/dokku
+ export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ export DOKKU_CONTAINER_LABEL=dokku
+ DOKKU_CONTAINER_LABEL=dokku
+ export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ parse_args ps:rebuild myapp
+ declare 'desc=top-level cli arg parser'
+ local next_index=1
+ local skip=false
+ args=("$@")
+ local args
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=2
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=3
+ return 0
+ args=("$@")
+ [[ ps:rebuild =~ ^--.* ]]
+ has_tty
+ declare 'desc=return 0 if we have a tty'
++ /usr/bin/tty
+ [[ /dev/pts/0 == \n\o\t\ \a\ \t\t\y ]]
+ return 0
++ id -un
+ [[ dokku != \d\o\k\k\u ]]
++ id -un
+ [[ dokku != \r\o\o\t ]]
+ [[ ps:rebuild =~ ^plugin:.* ]]
+ [[ -n '' ]]
+ dokku_auth ps:rebuild myapp
+ declare 'desc=calls user-auth plugin trigger'
+ export SSH_USER=root
+ SSH_USER=root
+ export SSH_NAME=default
+ SSH_NAME=default
+ plugn trigger user-auth root default ps:rebuild myapp
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ [[ ! -n '' ]]
+ return 0
+ case "$1" in
+ execute_dokku_cmd ps:rebuild myapp
+ declare 'desc=executes dokku sub-commands'
+ local PLUGIN_NAME=ps:rebuild
+ local PLUGIN_CMD=ps:rebuild
+ local implemented=0
+ local script
+ argv=("$@")
+ local argv
+ case "$PLUGIN_NAME" in
++ readlink -f /var/lib/dokku/plugins/enabled/ps
+ [[ /var/lib/dokku/core-plugins/available/ps == *core-plugins* ]]
+ [[ ps:rebuild == \p\s\:\r\e\b\u\i\l\d ]]
+ shift 1
+ [[ ! -z '' ]]
+ set -- ps:rebuild myapp
+ [[ -x /var/lib/dokku/plugins/enabled/ps:rebuild/subcommands/default ]]
+ [[ -x /var/lib/dokku/plugins/enabled/ps:rebuild/subcommands/ps:rebuild ]]
+ [[ -x /var/lib/dokku/plugins/enabled/ps/subcommands/rebuild ]]
+ /var/lib/dokku/plugins/enabled/ps/subcommands/rebuild ps:rebuild myapp
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ source /var/lib/dokku/plugins/available/ps/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++ source /var/lib/dokku/core-plugins/available/common/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
++ source /var/lib/dokku/plugins/available/config/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
+++ source /var/lib/dokku/core-plugins/available/common/functions
++++ set -eo pipefail
++++ [[ -n 1 ]]
++++ set -x
+ ps_rebuild_cmd ps:rebuild myapp
+ declare 'desc=rebuilds app via command line'
+ local cmd=ps:rebuild
+ [[ -z myapp ]]
+ ps_rebuild myapp
+ declare 'desc=rebuilds app from base image'
+ local APP=myapp
+ verify_app_name myapp
+ declare 'desc=verify app name format and app existence'
+ local APP=myapp
+ [[ ! -n myapp ]]
+ [[ ! myapp =~ ^[a-z].* ]]
+ [[ ! -d /home/dokku/myapp ]]
+ return 0
+ plugn trigger receive-app myapp
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ [[ ! -n '' ]]
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ git_receive_app myapp
+ declare 'desc=git receive-app plugin trigger'
+ local trigger=git_receive_app
+ local APP=myapp
+ local REV=
+ [[ ! -d /home/dokku/myapp/refs ]]
+ dokku git-build myapp
+ case "$(lsb_release -si)" in
++ lsb_release -si
+ export DOKKU_DISTRO=ubuntu
+ DOKKU_DISTRO=ubuntu
+ export DOKKU_IMAGE=gliderlabs/herokuish
+ DOKKU_IMAGE=gliderlabs/herokuish
+ export DOKKU_LIB_ROOT=/var/lib/dokku
+ DOKKU_LIB_ROOT=/var/lib/dokku
+ export PLUGIN_PATH=/var/lib/dokku/plugins
+ PLUGIN_PATH=/var/lib/dokku/plugins
+ export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ export DOKKU_API_VERSION=1
+ DOKKU_API_VERSION=1
+ export DOKKU_NOT_IMPLEMENTED_EXIT=10
+ DOKKU_NOT_IMPLEMENTED_EXIT=10
+ export DOKKU_VALID_EXIT=0
+ DOKKU_VALID_EXIT=0
+ export DOKKU_LOGS_DIR=/var/log/dokku
+ DOKKU_LOGS_DIR=/var/log/dokku
+ export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ export DOKKU_CONTAINER_LABEL=dokku
+ DOKKU_CONTAINER_LABEL=dokku
+ export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ parse_args git-build myapp
+ declare 'desc=top-level cli arg parser'
+ local next_index=1
+ local skip=false
+ args=("$@")
+ local args
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=2
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=3
+ return 0
+ args=("$@")
+ [[ git-build =~ ^--.* ]]
+ has_tty
+ declare 'desc=return 0 if we have a tty'
++ /usr/bin/tty
+ [[ /dev/pts/0 == \n\o\t\ \a\ \t\t\y ]]
+ return 0
++ id -un
+ [[ dokku != \d\o\k\k\u ]]
++ id -un
+ [[ dokku != \r\o\o\t ]]
+ [[ git-build =~ ^plugin:.* ]]
+ [[ -n '' ]]
+ dokku_auth git-build myapp
+ declare 'desc=calls user-auth plugin trigger'
+ export SSH_USER=root
+ SSH_USER=root
+ export SSH_NAME=default
+ SSH_NAME=default
+ plugn trigger user-auth root default git-build myapp
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ [[ ! -n '' ]]
+ return 0
+ case "$1" in
+ execute_dokku_cmd git-build myapp
+ declare 'desc=executes dokku sub-commands'
+ local PLUGIN_NAME=git-build
+ local PLUGIN_CMD=git-build
+ local implemented=0
+ local script
+ argv=("$@")
+ local argv
+ case "$PLUGIN_NAME" in
++ readlink -f /var/lib/dokku/plugins/enabled/git-build
+ [[ /var/lib/dokku/plugins/enabled/git-build == *core-plugins* ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-build/subcommands/default ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-build/subcommands/git-build ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-build/subcommands/git-build ]]
+ [[ 0 -eq 0 ]]
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/00_dokku-standard/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/20_events/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/apps/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/certs/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/checks/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/config/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/docker-options/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/domains/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/enter/commands git-build myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/git/commands git-build myapp
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ source /var/lib/dokku/plugins/available/apps/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++ source /var/lib/dokku/core-plugins/available/common/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
++ source /var/lib/dokku/plugins/available/config/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
+++ source /var/lib/dokku/core-plugins/available/common/functions
++++ set -eo pipefail
++++ [[ -n 1 ]]
++++ set -x
+ source /var/lib/dokku/plugins/available/config/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++ source /var/lib/dokku/core-plugins/available/common/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
+ case "$1" in
+ git_build_cmd git-build myapp
+ declare 'desc=lock git-build'
+ local cmd=git-build
+ local APP=myapp
+ local APP_BUILD_LOCK=/home/dokku/myapp/.build.lock
+ local 'APP_BUILD_LOCK_MSG=myapp is currently being deployed or locked. Waiting...'
++ flock -n /home/dokku/myapp/.build.lock true
++ echo 0
+ [[ 0 -ne 0 ]]
+ shift 1
+ flock -o /home/dokku/myapp/.build.lock dokku git-build-locked myapp
+ case "$(lsb_release -si)" in
++ lsb_release -si
+ export DOKKU_DISTRO=ubuntu
+ DOKKU_DISTRO=ubuntu
+ export DOKKU_IMAGE=gliderlabs/herokuish
+ DOKKU_IMAGE=gliderlabs/herokuish
+ export DOKKU_LIB_ROOT=/var/lib/dokku
+ DOKKU_LIB_ROOT=/var/lib/dokku
+ export PLUGIN_PATH=/var/lib/dokku/plugins
+ PLUGIN_PATH=/var/lib/dokku/plugins
+ export PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ PLUGIN_AVAILABLE_PATH=/var/lib/dokku/plugins/available
+ export PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ PLUGIN_ENABLED_PATH=/var/lib/dokku/plugins/enabled
+ export PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ PLUGIN_CORE_PATH=/var/lib/dokku/core-plugins
+ export PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ PLUGIN_CORE_AVAILABLE_PATH=/var/lib/dokku/core-plugins/available
+ export PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ PLUGIN_CORE_ENABLED_PATH=/var/lib/dokku/core-plugins/enabled
+ export DOKKU_API_VERSION=1
+ DOKKU_API_VERSION=1
+ export DOKKU_NOT_IMPLEMENTED_EXIT=10
+ DOKKU_NOT_IMPLEMENTED_EXIT=10
+ export DOKKU_VALID_EXIT=0
+ DOKKU_VALID_EXIT=0
+ export DOKKU_LOGS_DIR=/var/log/dokku
+ DOKKU_LOGS_DIR=/var/log/dokku
+ export DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ DOKKU_EVENTS_LOGFILE=/var/log/dokku/events.log
+ export DOKKU_CONTAINER_LABEL=dokku
+ DOKKU_CONTAINER_LABEL=dokku
+ export DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ DOKKU_GLOBAL_RUN_ARGS=--label=dokku
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ parse_args git-build-locked myapp
+ declare 'desc=top-level cli arg parser'
+ local next_index=1
+ local skip=false
+ args=("$@")
+ local args
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=2
+ for arg in '"$@"'
+ false
+ case "$arg" in
+ local next_index=3
+ return 0
+ args=("$@")
+ [[ git-build-locked =~ ^--.* ]]
+ has_tty
+ declare 'desc=return 0 if we have a tty'
++ /usr/bin/tty
+ [[ /dev/pts/0 == \n\o\t\ \a\ \t\t\y ]]
+ return 0
++ id -un
+ [[ dokku != \d\o\k\k\u ]]
++ id -un
+ [[ dokku != \r\o\o\t ]]
+ [[ git-build-locked =~ ^plugin:.* ]]
+ [[ -n '' ]]
+ dokku_auth git-build-locked myapp
+ declare 'desc=calls user-auth plugin trigger'
+ export SSH_USER=root
+ SSH_USER=root
+ export SSH_NAME=default
+ SSH_NAME=default
+ plugn trigger user-auth root default git-build-locked myapp
+ source /var/lib/dokku/core-plugins/available/common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ [[ ! -n '' ]]
+ return 0
+ case "$1" in
+ execute_dokku_cmd git-build-locked myapp
+ declare 'desc=executes dokku sub-commands'
+ local PLUGIN_NAME=git-build-locked
+ local PLUGIN_CMD=git-build-locked
+ local implemented=0
+ local script
+ argv=("$@")
+ local argv
+ case "$PLUGIN_NAME" in
++ readlink -f /var/lib/dokku/plugins/enabled/git-build-locked
+ [[ /var/lib/dokku/plugins/enabled/git-build-locked == *core-plugins* ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-build-locked/subcommands/default ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-build-locked/subcommands/git-build-locked ]]
+ [[ -x /var/lib/dokku/plugins/enabled/git-build-locked/subcommands/git-build-locked ]]
+ [[ 0 -eq 0 ]]
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/00_dokku-standard/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/20_events/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/apps/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/certs/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/checks/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/config/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/docker-options/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/domains/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/enabled/enter/commands git-build-locked myapp
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_ENABLED_PATH/*/commands'
+ set +e
```
Can you remove the redirect to stderr on [this line](https://github.com/dokku/dokku/blob/a9a9d0a898c23a1df9252b6ae1f8fecc2ff1be4e/plugins/git/functions#L57) and try pushing again?
Ok, I get this:
```
remote: + git fetch --depth=1 origin refs/tags/dokku/b4287d41e948d5bd423edd1d5528d86784678d8c
remote: fatal: write error: No space left on device
```
But I have a different version of that git file you linked to, it is not called `functions` but `commands` (`dokku/plugins/enabled/git/commands`).
Okay, so even though I cleaned up there is no space. Hm. Any idea how I can make more space? Most of the space is taken in /var/lib/docker/aufs/diff.
It is a 30 GB server and there is only one app. :)
Here is the disk space info if that helps:
```
Filesystem Size Used Avail Use% Mounted on
udev 487M 4.0K 487M 1% /dev
tmpfs 100M 552K 99M 1% /run
/dev/disk/by-label/DOROOT 30G 27G 1.4G 96% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 497M 2.8M 495M 1% /run/shm
none 100M 0 100M 0% /run/user
overflow 1.0M 0 1.0M 0% /tmp
none 30G 27G 1.4G 96% /var/lib/docker/aufs/mnt/6cf7122027c0fc8f30350803dc83d016398b69bf17e5b69349b8559718d787f6
shm 64M 0 64M 0% /var/lib/docker/containers/5d0b9c824ea6a7b05c73293f5ac92e8bc6060cd9b69d48aef2a90555a6124a4c/shm
none 30G 27G 1.4G 96% /var/lib/docker/aufs/mnt/96c252a39a50a42b7f30b641c4c35bf3fe4fc339990a7127d7a7f6a83aad4e75
shm 64M 0 64M0% /var/lib/docker/containers/f149b478793abf788bb3c13b89509965e678083db541cc7a762059dcbcdb2737/shm
none 30G 27G 1.4G 96% /var/lib/docker/aufs/mnt/6a54418da4a675d48752d2bfc679ea3d72b30fd744732450d3208931edfea689
shm 64M 4.0K 64M 1% /var/lib/docker/containers/0e442500e4008b22e8247296a9a79e67fe4aeac3526dc265fa1fa633e77ef77b/shm
none 30G 27G 1.4G 96% /var/lib/docker/aufs/mnt/947a39aeaadd4774c45360bb6bd792ca94cbae4570e29147fad213e188137e70
shm 64M 0 64M 0% /var/lib/docker/containers/64af2fc33af08a1e7907d5ff2f419a5187c8c170eace4f6a08a65ac6484d3e53/shm
none 30G 27G 1.4G 96% /var/lib/docker/aufs/mnt/163a716c9ff437e0dc48aea195b5907cb78e820aa4838a65efb7c8849a66e361
shm 64M 0 64M 0% /var/lib/docker/containers/3cb07b2cd8962f3c9e27d032e3bdd033bcfb4e723f943a5cc2193907ba662919/shm
```空orphaned diffs
I'd like to know why docker uses so much disk, even after removing _all_ containers, images, and volumes.
It looks like this "diff" has a layer, but the layer isn't referenced by anything at all.
```
/var/lib/docker/aufs/diff# du-summary
806628 c245c4c6d71ecdd834974e1e679506d33c4aac5f552cb4b28e727a596efc1695-removing
302312 a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
302304 957e78f9f9f4036689734df16dabccb98973e2c3de0863ef3f84de85dca8d92d
302256 8db1d610f3fbc71415f534a5d88318bbd2f3f783375813f2288d15f15846d312
288204 ac6b8ff4c0e7b91230ebf4c1caf16f06c1fdceff6111fd58f4ea50fc2dd5050b
288180 04a478c413ea80bcfa7f6560763beef991696eace2624254479e5e5dd69708c6
287804 d033ab6e4e5231dc46c6c417c680b239bb0e843024738517cbb0397128e166ca
233420 8e21143dca49e30cae7475b71b5aee9b92abe2069fbb9ab98ce9c334e3f6d4fa
212668 a631b94f7a2d5d21a96a78e9574d39cdeebbc81b51ac6c58bd48dc4045656477
205120 ae13341f8c08a925a95e5306ac039b0e0bbf000dda1a60afb3d15c838e43e349
205120 8d42279017d6095bab8d533ab0f1f7de229aa7483370ef53ead71fe5be3f1284
205116 59b3acd8e0cfd194d44313978d4b3769905cdb5204a590069c665423b10150e3
205116 040af0eee742ec9fb2dbeb32446ce44829cd72f02a2cf31283fcd067e73798ab
158024 ef0a29ff0b515c8c57fe78bcbd597243de9f7b274d9b212c774d91bd45a6c9b1
114588 061bd7e021afd4aaffa9fe6a6de491e10d8d37d9cbe7612138f58543e0985280
114576 149e8d2745f6684bc2106218711991449c452d4c7e6203e2a0f46651399162b0
114532 52b28112913abb0ed1b3267a0baa1cacd022ca6611812d0a8a428e61ec399589
114300 52475beba19687a886cba4bdb8508d5aaf051ceb52fb3a65294141ab846c8294
76668 4e6afb958b5ee6dea6d1a886d19fc9c780d4ecc4baeebfbde31f9bb97732d10d
76640 c61340c6a962ddd484512651046a676dbbc6a5d46aecc26995c49fe987bf9cdc
/var/lib/docker/aufs/diff# du -hs a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
296M a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
$ docker-find a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
+ docker=/var/lib/docker
+ sudo find /var/lib/docker '(' -path '/var/lib/docker/aufs/diff/*' -o -path '/var/lib/docker/aufs/mnt/*' ')' -prune -o -print
+ grep a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
/var/lib/docker/aufs/layers/a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
+ sudo find /var/lib/docker '(' -path '/var/lib/docker/aufs/diff/*' -o -path '/var/lib/docker/aufs/mnt/*' ')' -prune -o -type f -print0
+ sudo xargs -0 -P20 grep -l a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
/var/lib/docker/aufs/layers/993e4988c510ec3ab4f6d139740a059df40585576f8196817e573a9684554c5c
/var/lib/docker/aufs/layers/95e68d59a8704f2bb52cc1306ca910ddb7af8956eb7c57970fcf7d8b3d9baddb
/var/lib/docker/aufs/layers/4e6afb958b5ee6dea6d1a886d19fc9c780d4ecc4baeebfbde31f9bb97732d10d
/var/lib/docker/aufs/layers/fd895b6f56aedf09c48dba97931a34cea863a21175450c31b6ceadde03f7b3da
/var/lib/docker/aufs/layers/ac6b8ff4c0e7b91230ebf4c1caf16f06c1fdceff6111fd58f4ea50fc2dd5050b
/var/lib/docker/aufs/layers/f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579-init
/var/lib/docker/aufs/layers/d5bbef5adf2efb6f15d4f96c4bee21beb955255d1ec17baf35de66e98e6c7328
/var/lib/docker/aufs/layers/9646360df378b88eae6f1d6288439eebd9647d5b9e8a471840d4a9d6ed5d92a4
/var/lib/docker/aufs/layers/cf9fd1c4a64baa39b6d6d9dac048ad2fff3c3fe13924b07377e767eed230ba9f
/var/lib/docker/aufs/layers/f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579
/var/lib/docker/aufs/layers/23ce5a473b101d85f0e9465debe5a0f3b8a2079b99528a797b02052d06bc11d8
/var/lib/docker/image/aufs/layerdb/sha256/d1c659b8e3d0e893e95c8eedc755adcb91a1c2022e1090376b451f7206f9b1c0/cache-id
$ sudo cat /var/lib/docker/image/aufs/layerdb/sha256/d1c659b8e3d0e893e95c8eedc755adcb91a1c2022e1090376b451f7206f9b1c0/diff
sha256:b5185949ba02a6e065079660b0536672c9691fb0e0cb1fd912b2c7b29c91d625
$ docker-find sha256:b5185949ba02a6e065079660b0536672c9691fb0e0cb1fd912b2c7b29c91d625
+ docker=/var/lib/docker
+ sudo find /var/lib/docker '(' -path '/var/lib/docker/aufs/diff/*' -o -path '/var/lib/docker/aufs/mnt/*' ')' -prune -o -print
+ grep sha256:b5185949ba02a6e065079660b0536672c9691fb0e0cb1fd912b2c7b29c91d625
+ sudo find /var/lib/docker '(' -path '/var/lib/docker/aufs/diff/*' -o -path '/var/lib/docker/aufs/mnt/*' ')' -prune -o -type f -print0
+ sudo xargs -0 -P20 grep -l sha256:b5185949ba02a6e065079660b0536672c9691fb0e0cb1fd912b2c7b29c91d625
/var/lib/docker/image/aufs/layerdb/sha256/d1c659b8e3d0e893e95c8eedc755adcb91a1c2022e1090376b451f7206f9b1c0/diff
```
```
# docker --version
Docker version 1.10.3, build 99b71ce
# docker info
Containers: 3
Running: 0
Paused: 0
Stopped: 3
Images: 29
Server Version: 1.10.3
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 99
Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 3.13.0-83-generic
Operating System: |unknown|
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 125.9 GiB
Name: dev34-devc
ID: VKMX:YMJ2:3NGV:5J6I:5RYM:AVBK:QPOZ:ODYE:VQ2D:AF2J:2LEM:TKTE
WARNING: No swap limit support
```
I should also show that docker lists no containers, volumes, or images:
```
$ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ docker volume ls
DRIVER VOLUME NAME
```
strange; especially because of;
```
Containers: 3
Running: 0
Paused: 0
Stopped: 3
Images: 29
```
which doesn't match the output of `docker images` / `docker ps`.
What operating system are you running on?
```
Operating System: |unknown|
```
@tonistiigi any idea?
That was afterward. I guess some processes kicked off in the meantime.
The state I'm referring to (I have now) is:
```
$ docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
```
And I still have:
```
$ sudo du -hs /var/lib/docker/aufs/diff/a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
296M /var/lib/docker/aufs/diff/a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
```
We're on Ubuntu Lucid with an upgraded kernel =/
```
$ uname -a
Linux dev34-devc 3.13.0-83-generic #127-Ubuntu SMP Fri Mar 11 00:25:37 UTC 2016 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 10.04.1 LTS
Release: 10.04
Codename: lucid
```
It seems an interesting issue.
is it possible to have way to reproduce it ? @bukzor
Surely it's possible, but I don't know how.
Please try running the below script on one of your active docker hosts and see what's left.
In our case, there's always plenty of diffs left behind.
``` #!bash
#!/bin/bash
set -eu
echo "WARNING:: This will stop ALL docker processes and remove ALL docker images."
read -p "Continue (y/n)? "
if [ "$REPLY" != "y" ]; then
echo "Aborting."
exit 1
fi
xdocker() { exec xargs -P10 -r -n1 --verbose docker "$@"; }
set -x
# remove containers
docker ps -q | xdocker stop
docker ps -aq | xdocker rm
# remove tags
docker images | sed 1d | grep -v '^|none|' | col 1 2 | sed 's/ /:/' | xdocker rmi
# remove images
docker images -q | xdocker rmi
docker images -aq | xdocker rmi
# remove volumes
docker volume ls -q | xdocker volume rm
```
One possible way I see this happening is that if there are errors on aufs unmounting. For example, if there are EBUSY errors then probably the image configuration has already been deleted before.
@bukzor Would be very interesting if there was a reproducer that would start from an empty graph directory, pull/run images and get it into a state where it doesn't fully clean up after running your script.
That would be interesting, but sounds like a full day's work.
I can't commit to that.
Here's some more data regarding the (arbitrarily selected) troublesome diff above, `a800`.
``` #!sh
$ docker-find a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea | sudo xargs -n1 wc -l | sort -rn
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -print
+ grep a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -type f -print0
+ sudo xargs -0 -P20 grep -l a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
15 /nail/var/lib/docker/aufs/layers/f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579
14 /nail/var/lib/docker/aufs/layers/f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579-init
13 /nail/var/lib/docker/aufs/layers/993e4988c510ec3ab4f6d139740a059df40585576f8196817e573a9684554c5c
12 /nail/var/lib/docker/aufs/layers/cf9fd1c4a64baa39b6d6d9dac048ad2fff3c3fe13924b07377e767eed230ba9f
11 /nail/var/lib/docker/aufs/layers/4e6afb958b5ee6dea6d1a886d19fc9c780d4ecc4baeebfbde31f9bb97732d10d
10 /nail/var/lib/docker/aufs/layers/23ce5a473b101d85f0e9465debe5a0f3b8a2079b99528a797b02052d06bc11d8
9 /nail/var/lib/docker/aufs/layers/95e68d59a8704f2bb52cc1306ca910ddb7af8956eb7c57970fcf7d8b3d9baddb
8 /nail/var/lib/docker/aufs/layers/ac6b8ff4c0e7b91230ebf4c1caf16f06c1fdceff6111fd58f4ea50fc2dd5050b
7 /nail/var/lib/docker/aufs/layers/fd895b6f56aedf09c48dba97931a34cea863a21175450c31b6ceadde03f7b3da
6 /nail/var/lib/docker/aufs/layers/d5bbef5adf2efb6f15d4f96c4bee21beb955255d1ec17baf35de66e98e6c7328
5 /nail/var/lib/docker/aufs/layers/9646360df378b88eae6f1d6288439eebd9647d5b9e8a471840d4a9d6ed5d92a4
4 /nail/var/lib/docker/aufs/layers/a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
0 /nail/var/lib/docker/image/aufs/layerdb/sha256/d1c659b8e3d0e893e95c8eedc755adcb91a1c2022e1090376b451f7206f9b1c0/cache-id
```
So we see there's a chain of child layers, with `f3286009193` as the tip.
```
$ docker-find f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579'$'
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -print
+ grep --color 'f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579$'
/nail/var/lib/docker/aufs/layers/f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -type f -print0
+ sudo xargs -0 -P20 grep --color -l 'f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579$'
/nail/var/lib/docker/image/aufs/layerdb/mounts/eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e/mount-id
```
So that layer was used in mount `eb809c0321`. I don't find any references to that mount anywhere:
```
$ docker-find eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -print
+ grep --color eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e
/nail/var/lib/docker/image/aufs/layerdb/mounts/eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e
/nail/var/lib/docker/image/aufs/layerdb/mounts/eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e/mount-id
/nail/var/lib/docker/image/aufs/layerdb/mounts/eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e/init-id
/nail/var/lib/docker/image/aufs/layerdb/mounts/eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e/parent
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -type f -print0
+ sudo xargs -0 -P20 grep --color -l eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e
```
Is there any way to find what container that mount was used for?
The doc only says the mount ID is no longer equal to the container ID, which isn't very helpful.
https://docs.docker.com/engine/userguide/storagedriver/aufs-driver/
@bukzor `eb809c0321` is the container id. What docs mean is that aufs id (`f3286009193f` in your case) is not container id.
/cc @dmcgowan as well
@tonistiigi OK.
Then obviously the mount has outlived its container.
At what point in the container lifecycle is the mount cleaned up?
Is this the temporary writable aufs for running/stopped containers?
@bukzor (rw) mount is deleted on container deletion. Unmount happens on container process stop. Diff folders are a place where the individual layer contents are stored, it doesn't matter is the layer is mounted or not.
@bukzor The link between the aufs id and container id can be found at `image/aufs/layerdb/mounts/|container-id|/mount-id`. From just knowing an aufs id the easiest way to find the container id is to grep the `image/aufs/layerdb` directory for it. If nothing is found, then the cleanup was not completed cleanly.
Running into similar issue.
We're running daily CI in the docker daemon server. /var/lib/docker/aufs/diff takes quite a mount of disk capacity, which it shouldn't be.
Still `2gb` in `aufs/diff` after trying everything reasonable suggested here or in related threads (including @bukzor's bash script above).
Short of a proper fix, is there any straightforward way of removing the leftover mounts without removing all other images at the same time? (If no containers are running currently, I guess there should be no mounts, right?)
I am experiencing the same issue. I am using this machine to test a lot of containers, then commit/delete. My /var/lib/docker/aufs directory is currently 7.9G heavy. I'm going to have to move this directory to another mount point, because storage on this one is limited. :(
```
# du -sh /var/lib/docker/aufs/diff/
1.9T /var/lib/docker/aufs/diff/
```
@mcallaway Everything in `aufs/diff` is going to be fs writes performed in a container.
I have the same issue. All containers which I have are in running state, but there are lots of aufs diff directories which don't relate to these containers and relate to old removed containers. I can remove them manually, but it is not an option. There should be a reason for such a behavior.
I use k8s 1.3.5 and docker 1.12.
Running of the `docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc spotify/docker-gc` helped.
I have the same issue. I'm using Gitlab CI with dind (docker in docker).
IMHO when image in the registry was updated within the same tag and it was pulled, then related container was restarted, old container and image are not GCed unless you run `spotify/docker-gc`.
Can someone else confirm this?
@kayrus correct, docker will not automatically assume that an "untagged" image should also be _removed_. Containers could still be using that image, and you can still start new containers from that image (referencing it by its ID). You can remove "dangling" images using `docker rmi $(docker images -qa -f dangling=true)`. Also, docker 1.13 will get data management commands (see https://github.com/docker/docker/pull/26108), which allow you to more easily cleanup unused images, containers, etc.
@thaJeztah does `/var/lib/docker/aufs/diff/` actually contain the "untagged" images?
@kayrus yes they are part of the images (tagged, and untagged)
getting a similar issue, no containers/images/volumes, ~13Gb of diffs
```
$ docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 1030
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-32-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.861 GiB
Name: gitrunner
ID: GSAW:6X5Z:SHHU:NZIM:O76D:P5OE:7OZG:UFGQ:BOAJ:HJFM:5G6W:5APP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
```
```
$ docker volume ls
DRIVER VOLUME NAME
$ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$
```
```
$ df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/mapper/gitrunner--docker-lib--docker 18G 15G 2.6G 85% /var/lib/docker
```
```
/var/lib/docker# sudo du -sm aufs/*
13782 aufs/diff
5 aufs/layers
5 aufs/mnt
```
``` shell
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 1.12.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: xfs
Dirs: 1122
```
Same issue here. I understand 1.13 may get data management commands but in the meantime, I just want to safely delete the contents of this directory without killing Docker.
This is relatively blocking at this point.
Same here. Still no official solution?
I've brought this up a few different times in (Docker Community) Slack. Each time a handful of people run through a list of garbage collection scripts/cmds I should run as a solution.
While those have helped (read: not solved - space is still creeping towards full) in the interim, I think we can all agree that's not the ideal long term fix.
@jadametz 1.13 has `docker system prune`.
Beyond that, I'm not sure how else Docker can help (open to suggestion). The images aren't just getting to the system on their own, but rather through pulls, builds, etc.
In terms of actual orphaned layers (no images on the system referencing them), we'll need to address that separately.
I have exactly the same issue!
```docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 2501
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-96-generic
Operating System: Ubuntu 14.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 14.69 GiB
Name: ip-172-31-45-4
ID: R5WV:BXU5:AV6T:GZUK:SAEA:6E74:PRSO:NQOH:EPMQ:W6UT:5DU4:LE64
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
```
No images, containers or volumes. 42Gb in aufs/diff
Anything to help clear this directory safely would be very useful! Tried everything in this thread without any success. Thanks.
@adamdry only third-party script: https://github.com/docker/docker/issues/22207#issuecomment-252560212
Thanks @kayrus I did indeed try that and it increased my total disk usage slightly and didn't appear to do anything to the aufs/diff directory.
I also tried `docker system prune` which didn't run. And I tried `docker rmi $(docker images -qa -f dangling=true)` which didn't find any images to remove.
For anyone interested I'm now using this to clean down all containers, images, volumes and old aufs:
`### FYI I am a Docker noob so I don't know if this causes any underlying issues but it does work for me - use at your own risk ###`
Lots of inspiration taken from here: http://stackoverflow.com/questions/30984569/error-error-creating-aufs-mount-to-when-building-dockerfile
```
docker rm -f $(docker ps -a -q) || docker rmi -f $(docker images -q) || docker rmi -f $(docker images -a -q)
service docker stop
rm -rf /var/lib/docker/aufs
rm -rf /var/lib/docker/image/aufs
rm -f /var/lib/docker/linkgraph.db
service docker start
```
@adamdry Best to not use `-f` when doing rm/rmi as it will hide errors in removal.
I do consider the current situation... where `-f` hides an error and then we are left with some left-over state that is completely invisible to the user... as a bug.
I'm also seeing this on a completely new and unsurprising installation:
```
root@builder:/var/lib/docker# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.4
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 63
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: overlay host null bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.625 GiB
Name: builder
ID: 2WXZ:BT74:G2FH:W7XD:VVXM:74YS:EA3A:ZQUK:LPID:WYKF:HDWC:UKMJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Insecure Registries:
127.0.0.0/8
root@builder:/var/lib/docker# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
root@builder:/var/lib/docker# docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
root@builder:/var/lib/docker# du -hd2
4.0K ./swarm
6.0M ./image/aufs
6.0M ./image
4.0K ./trust
28K ./volumes
4.0K ./containers
276K ./aufs/layers
292K ./aufs/mnt
1.5G ./aufs/diff |-------------------------
1.5G ./aufs
4.0K ./tmp
72K ./network/files
76K ./network
1.5G .
root@builder:/var/lib/docker#
```
@robhaswell Seeing as it's a new install, do you want to try this? https://github.com/docker/docker/issues/22207#issuecomment-266784433
@adamdry I've already deleted `/var/lib/docker/aufs` as it was blocking my work. What do you expect your instructions to achieve? If they stop the problem from happening again in the future, then I can try to recreate the issue and try your instructions. However if the purpose is just to free up the space then I've already achieved that.
@robhaswell Yeah it was to free up disk space but I had follow up issues when trying to rebuild my images but by following all the steps in that script it resolved those issues.
During build, if the build process is interrupted during layer build process (which also contains a blob to be copied), followed by stopping the container, it leaves behind data in /var/lib/docker/aufs/diff/. It showed up a dangling image. Cleaning it up too didn't release the space. Is it possible to include it as a part of docker system prune? Only deleting the blob data inside this folder frees up the space, which I am not sure will cause any issue or not.
Docker version : 1.13.0-rc1
| During build, if the build process is interrupted during layer build process (which also contains a blob to be copied), followed by stopping the container, it leaves behind data
This could also be the cause of my problems - I interrupt a lot of builds.
During docker pull, observed the following two cases:
1. if the process is interrupted when it says download (which downloads the image layer in /var/lib/docker/tmp/) , it cleans up all the data in that folder
2. If the process is interrupted when it says extracting(which i suppose is extracting the layer from tmp to /var/lib/docker/aufs/diff/), it cleans up the tmp and diff blob data both.
During image build process,
1. On interrupting the process when "Sending build context to docker daemon" ( which copies blob data in my case in /var/lib/docker/tmp/), it remains there forever and cannot be cleaned by any command except manually deleting it. I am not sure how the apt get updates in image are handled.
2. While the layer is being built which contains a blob data, say a large software setup, if the process is interrupted, the docker container keeps working on the image. In my case only 1 layer blob data, which is already available in tmp folder, makes up the whole image. But, if the container is stopped using the docker stop command, two cases happen:
a. if the mount process is still happening, it will leave behind data in tmp and diff folder.
b. If the data copied in diff folder, it will remove the data from tmp folder and leave data in diff folder and maybe mount folder.
We have an automated build process, which needs a control to stop any build process gracefully. Recently, a process got killed by kernel due to out of memory error on a machine which was of low configuration.
If one image is to be built of 2 layers, 1 layer is built and 2nd is interrupted, Docker system prune seems to clean up the data for the container of layer which was interrupted and container stopped. But it doesn't clean up the data of the previous layers in case of interrupt. Also, it didn't reflect the total disk space claimed. Ran these tests on AWS, ubuntu 14.04, x86_64 bit system with aufs filesystem. Ran the docker prune test with docker 1.13.0 rc3 and docker 1.12
@thaJeztah
Please let me know if i am misinterpreting anything.
I opened an issue for the `/var/lib/docker/tmp` files not being cleaned up; https://github.com/docker/docker/issues/29486
| Docker system prune seems to clean up the data for the container of layer which was interrupted and container stopped. But it doesn't clean up the data of the previous layers in case of interrupt.
I tried to reproduce that situation, but wasn't able to see that with a simple case;
Start with a clean install empty `/var/lib/docker`, create a big file for
testing, and a Dockerfile;
```bash
mkdir repro || cd repro
fallocate -l 300M bigfile
```
```Dockerfile
cat | Dockerfile ||EOF
FROM scratch
COPY ./bigfile /
COPY ./bigfile /again/
COPY ./bigfile /and-again/
EOF
```
start `docker build`, and cancel while building, but _after_ the build
context has been sent;
```bash
docker build -t stopme .
Sending build context to Docker daemon 314.6 MB
Step 1/4 : FROM scratch
---|
Step 2/4 : COPY ./bigfile /
---| 28eb6d7b0920
Removing intermediate container 98876b1673bf
Step 3/4 : COPY ./bigfile /again/
^C
```
check content of `/var/lib/docker/aufs/`
```bash
du -h /var/lib/docker/aufs/
301M /var/lib/docker/aufs/diff/9127644c356579741348f7f11f50c50c9a40e0120682782dab55614189e82917
301M /var/lib/docker/aufs/diff/81fd6b2c0cf9a28026cf8982331016a6cd62b7df5a3cf99182e7e09fe0d2f084/again
301M /var/lib/docker/aufs/diff/81fd6b2c0cf9a28026cf8982331016a6cd62b7df5a3cf99182e7e09fe0d2f084
601M /var/lib/docker/aufs/diff
8.0K /var/lib/docker/aufs/layers
4.0K /var/lib/docker/aufs/mnt/9127644c356579741348f7f11f50c50c9a40e0120682782dab55614189e82917
4.0K /var/lib/docker/aufs/mnt/81fd6b2c0cf9a28026cf8982331016a6cd62b7df5a3cf99182e7e09fe0d2f084
4.0K /var/lib/docker/aufs/mnt/b6ffb1d5ece015ed4d3cf847cdc50121c70dc1311e42a8f76ae8e35fa5250ad3-init
16K /var/lib/docker/aufs/mnt
601M /var/lib/docker/aufs/
```
run the `docker system prune` command to clean up images, containers;
```
docker system prune -a
WARNING! This will remove:
- all stopped containers
- all volumes not used by at least one container
- all networks not used by at least one container
- all images without at least one container associated to them
Are you sure you want to continue? [y/N] y
Deleted Images:
deleted: sha256:253b2968c0b9daaa81a58f2a04e4bc37f1dbf958e565a42094b92e3a02c7b115
deleted: sha256:cad1de5fd349865ae10bfaa820bea3a9a9f000482571a987c8b2b69d7aa1c997
deleted: sha256:28eb6d7b09201d58c8a0e2b861712701cf522f4844cf80e61b4aa4478118c5ab
deleted: sha256:3cda5a28d6953622d6a363bfaa3b6dbda57b789e745c90e039d9fc8a729740db
Total reclaimed space: 629.1 MB
```
check content of `/var/lib/docker/aufs/`
```bash
du -h /var/lib/docker/aufs/
4.0K /var/lib/docker/aufs/diff
4.0K /var/lib/docker/aufs/layers
4.0K /var/lib/docker/aufs/mnt/b6ffb1d5ece015ed4d3cf847cdc50121c70dc1311e42a8f76ae8e35fa5250ad3-init
8.0K /var/lib/docker/aufs/mnt
20K /var/lib/docker/aufs/
```
I do see that the `-init` mount is left behind, I'll check if we can solve
that (although it's just an empty directory)
The only difference in the dockerfile I had used was ( to create different layers)
FROM scratch
COPY ["./bigfile", "randomNoFile1", /]
COPY ["./bigfile", "randomNoFile2", /]
EOF
I am not sure if it makes a difference.
No, the problem isn't about the empty init folders. In my case, it was te blob. However, i can recheck it on monday and update.
Also, was using a 5GB file, created it by reading bytes from dev urandom.
In your case, the same file is added 2 times. Would that create single layer and mount the 2nd layer from it or would it be 2 separate layers? In my case, its always 2 separate layers.
@thaJeztah
Thank you for such a quick response on the issue. Addition of this feature would be of great help!
@monikakatiyar16 I tried to reproduce this as well with canceling the build multiple times during both `ADD` and `RUN` commands but couldn't get anything to leak to `aufs/diff` after deletion. I couldn't quite understand what container you are stopping because containers should not be running during `ADD/COPY` operations. If you can put together a reproducer that we could run that would be greatly appreciated.
Its possible that I could be doing something wrong. Since I am travelling on weekend, I will reproduce it and update all the required info here on Monday.
@tonistiigi @thaJeztah
I feel you are right. There are actually no containers that are listed as active and running. Instead there are dead containers. Docker system prune didn't work in my case, might be since the process didn't get killed with Ctrl+C. Instead, it kept running at the background. In my case, that would be the reason, it couldn't remove those blobs.
When I interrupt the process using Ctrl+C, the build process gets killed, but a process for docker-untar remains alive in the background, which keeps working on building the image. (Note: /var/lib/docker is soft linked to /home/lib/docker to use the EBS volumes for large data on AWS)
`root 12700 10781 7 11:43 ? 00:00:04 docker-untar /home/lib/docker/aufs/mnt/d446d4f8a7dbae162e7578af0d33ac38a63b4892905aa86a8d131c1e75e2828c`
I have attached the script I had been using for creating large files and building the image (gc_maxpush_pull.sh)
Also attached the behaviour of build process for building an image-interrupting it with Ctrl+C (DockerBuild_WOProcessKill) and building image -interrupting it with Ctrl+C - killing the process (DockerBuild_WithProcessKill)
Using the commands -
To create large file : `./gc_maxpush_pull.sh 1 5gblayer 0 512 1`
To build images : `./gc_maxpush_pull.sh 1 5gblayer 1 512 1`
[DockerBuild.zip](https://github.com/docker/docker/files/660942/DockerBuild.zip)
Steps to replicate :
1. Create a large file of 5GB
2. Start the build process and interrupt it only after Sending build context is over and it's actually copying the blob.
3. It completes building the image after a while and shows it up in docker images (as in case 1 attached by me - DockerBuild_WOProcessKill)
4. If the process is killed, it takes a while and leaves the blob data in /diff (which it should on killing process abruptly as attached in file - DockerBuild_WithProcessKill)
If what I am assuming is correct, then this might not be an issue with docker prune, instead with killing of docker build that is somehow not working for me.
Is there a graceful way of interrupting or stopping the build image process, which also takes care of cleaning up the copied data (as handled in docker pull)?
Previously, I was not killing the process. I am also curious what docker-untar does and why it mounts it to /mnt and /diff folders both and later clean out /mnt folder?
Tested this with Docker version 1.12.5, build 7392c3b on AWS
docker info
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 0
Server Version: 1.12.5
Storage Driver: aufs
Root Dir: /home/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 4
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: overlay bridge null host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-105-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859 GiB
Name: master
ID: 2NQU:D2C5:5WPL:IIDR:P6FO:OAG7:GHW6:ZJMQ:VDHI:B5CI:XFZJ:ZSZM
Docker Root Dir: /home/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
@monikakatiyar16 When I manually kill `untar` process during build I get `Error processing tar file(signal: killed):` in the build output. Leaving behind the container in `docker ps -a` is the correct behavior, same thing happens in any build error and lets you debug the problems that caused the build to fail. I have no problem with deleting this container though and if I do that all data in `/var/lib/docker/aufs` is cleaned up as well.
@tonistiigi Yes you are correct. I was able to delete the volume associated with the container and it cleaned up everything, after killing the docker-untar process. Docker system prune also works in this case.
The actual issue which left over volumes was the case, when without killing the docker-untar process, i tried removing the docker container along with the volumes - which gave the following error :
`docker rm -v -f $(docker ps -a -q)`
`Error response from daemon: Driver aufs failed to remove root filesystem 97931bf059a0ec219efd3f762dbb173cf9372761ff95746358c08e2b61f7ce79: rename /home/lib/docker/aufs/diff/359d27c5b608c9dda1170d1e34e5d6c5d90aa2e94826257f210b1442317fad70 /home/lib/docker/aufs/diff/359d27c5b608c9dda1170d1e34e5d6c5d90aa2e94826257f210b1442317fad70-removing: device or resource busy`
Daemon logs:
`Error removing mounted layer 78fb899aab981557bc2ee48e9738ff4c2fcf2d10a1984a62a77eefe980c68d4a: rename /home/lib/docker/aufs/diff/d2605125ef072de79dc948f678aa94dd6dde562f51a4c0bd08a210d5b2eba5ec /home/lib/docker/aufs/diff/d2605125ef072de79dc948f678aa94dd6dde562f51a4c0bd08a210d5b2eba5ec-removing: device or resource busy
ERRO[0956] Handler for DELETE /v1.25/containers/78fb899aab98 returned error: Driver aufs failed to remove root filesystem 78fb899aab981557bc2ee48e9738ff4c2fcf2d10a1984a62a77eefe980c68d4a: rename /home/lib/docker/aufs/diff/d2605125ef072de79dc948f678aa94dd6dde562f51a4c0bd08a210d5b2eba5ec /home/lib/docker/aufs/diff/d2605125ef072de79dc948f678aa94dd6dde562f51a4c0bd08a210d5b2eba5ec-removing: device or resource busy
ERRO[1028] Error unmounting container 78fb899aab981557bc2ee48e9738ff4c2fcf2d10a1984a62a77eefe980c68d4a: no such file or directory`
It seems that the order right now to be followed to interrupt a docker build is :
`Interrupt docker build | Kill docker untar process | remove container and volume : docker rm -v -f $(docker ps -a -q)`
For `docker v1.13.0-rc4`, it can be `Interrupt docker build | Kill docker untar process | docker system prune -a`
This seems to work perfectly. There are no issues of cleanup, instead the only issue is the docker-untar process not being killed along with the docker-build process.
I will search/update/log a new issue for graceful interrupt of docker build that also stops docker-untar process along with it.
(Verified this with docker v1.12.5 and v1.13.0-rc4)
Update : On killing docker-untar while Sending build context to docker daemon, it gives an error in build : `Error response from daemon: Error processing tar file(signal: terminated)` , but during layer copy it doesn't (for me)
Thanks for being so patient and for giving your time!
I'm seeing `/var/lib/docker/aufs` consistently increase in size on a docker swarm mode worker. This thing is mostly autonomous being managed by the swarm manager and very little manual container creation aside from some maintenance commands here and there.
I do run `docker exec` on service containers; not sure if that may be a cause.
My workaround to get this resolved in my case was to start up another worker, set the full node to `--availability=drain` and manually move over a couple of volume mounts.
```
ubuntu@ip-172-31-18-156:~$ docker --version
Docker version 1.12.3, build 6b644ec
```
This has hit our CI server for ages. This needs to be fixed.
@orf thanks
Same issue here. Neither container, volumes and image removing, nore Docker 1.13 cleanup commands have any effect.
I also confirm I did some image build cancels. Maybe that leaves folders that can't be reacher either.
I'll use the good old rm method for now, but this is clearly a bug.
Files in the /var/lib/docker/aufs/diff fills up 100% space for /dev/sda1 filesystem of 30G
root@Ubuntu:/var/lib/docker/aufs/diff# df -h
Filesystem Size Used Avail Use% Mounted on
udev 14G 0 14G 0% /dev
tmpfs 2.8G 273M 2.5G 10% /run
**/dev/sda1 29G 29G 0 100% /**
tmpfs 14G 0 14G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 14G 0 14G 0% /sys/fs/cgroup
/dev/sdb1 197G 60M 187G 1% /mnt
tmpfs 2.8G 0 2.8G 0% /run/user/1000
du -h -d 1 /var/lib/docker/aufs/diff | grep '[0-9]G\|'
shows
4.1G /var/lib/docker/aufs/diff/a0cde42cbea362bbb2a73ffbf30059bcce7ef0256d1d7e186264f915d15
14G /var/lib/docker/aufs/diff/59aee33d8a607b5315ce103cd99f17b4dfdec73c9a2f3bb2afc7d02bfae
20G /var/lib/docker/aufs/diff
Also tried, **docker system prune**, that did not help.
Has anyone found a solution for this ongoing issue of super large files in diff before this bug is fixed in the code?
Yes, the method has already been given, but here is a apocalypse snippet that just destroy everything I put in place here at work (except local folders for the volumes). To put in the bashrc or another bash config file.
```
alias docker-full-cleanup='func_full-cleanup-docker'
func_full-cleanup-docker() {
echo "WARN: This will remove everything from docker: volumes, containers and images. Will you dare? [y/N] "
read choice
if [ \( "$choice" == "y" \) -o \( "$choice" == "Y" \) ]
then
sudo echo "| sudo rights check [OK]"
sizea=`sudo du -sh /var/lib/docker/aufs`
echo "Stopping all running containers"
containers=`docker ps -a -q`
if [ -n "$containers" ]
then
docker stop $containers
fi
echo "Removing all docker images and containers"
docker system prune -f
echo "Stopping Docker daemon"
sudo service docker stop
echo "Removing all leftovers in /var/lib/docker (bug #22207)"
sudo rm -rf /var/lib/docker/aufs
sudo rm -rf /var/lib/docker/image/aufs
sudo rm -f /var/lib/docker/linkgraph.db
echo "Starting Docker daemon"
sudo service docker start
sizeb=`sudo du -sh /var/lib/docker/aufs`
echo "Size before full cleanup:"
echo " $sizea"
echo "Size after full cleanup:"
echo " $sizeb"
fi
}```
I ran rm -rf command to remove the files from the diff folder for now. Probably will have to look into the script if the diff folder occupies the entire dis space again.
Hope to see this issue fixed in the code, instead of work arounds.
Hi, I have same issue in docker 1.10.2, I'm running kubernetes. this is my docker version:
```
Containers: 7
Running: 0
Paused: 0
Stopped: 7
Images: 4
Server Version: 1.10.2
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 50
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 4.4.0-31-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.954 GiB
Name: ubuntu-k8s-03
ID: NT23:5Y7J:N2UM:NA2W:2FHE:FNAS:56HF:WFFF:N2FR:O4T4:WAHC:I3PO
Debug mode (server): true
File Descriptors: 10
Goroutines: 23
System Time: 2017-02-14T15:25:00.740998058+09:00
EventsListeners: 0
Init SHA1: 3e247d0d32543488f6e70fbb7c806203f3841d1b
Init Path: /usr/lib/docker/dockerinit
Docker Root Dir: /var/lib/docker
WARNING: No swap limit support
```
I'm trying to track all unused diff directory under `/var/lib/docker/aufs/diff` and `/var/lib/docker/aufs/mnt/` by analyzing layer files under `/var/lib/docker/image/aufs/imagedb`, here is the script I used:
https://gist.github.com/justlaputa/a50908d4c935f39c39811aa5fa9fba33
But I met problem when I stop and restart the docker daemon, seems I make some inconsistent status of docker:
/var/log/upstart/docker.log:
```
DEBU[0277] Cleaning up old shm/mqueue mounts: start.
DEBU[0277] Cleaning up old shm/mqueue mounts: done.
DEBU[0277] Clean shutdown succeeded
Waiting for /var/run/docker.sock
DEBU[0000] docker group found. gid: 999
DEBU[0000] Server created for HTTP on unix (/var/run/docker.sock)
DEBU[0000] Using default logging driver json-file
INFO[0000] [graphdriver] using prior storage driver "aufs"
DEBU[0000] Using graph driver aufs
INFO[0000] Graph migration to content-addressability took 0.00 seconds
DEBU[0000] Option DefaultDriver: bridge
DEBU[0000] Option DefaultNetwork: bridge
INFO[0000] Firewalld running: false
DEBU[0000] /sbin/iptables, [--wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -D PREROUTING]
DEBU[0000] /sbin/iptables, [--wait -t nat -D OUTPUT]
DEBU[0000] /sbin/iptables, [--wait -t nat -F DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -X DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -F DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -X DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION]
DEBU[0000] /sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION]
DEBU[0000] /sbin/iptables, [--wait -t nat -n -L DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -N DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -n -L DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -n -L DOCKER-ISOLATION]
DEBU[0000] /sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION -j RETURN]
DEBU[0000] /sbin/iptables, [--wait -I DOCKER-ISOLATION -j RETURN]
/var/run/docker.sock is up
DEBU[0000] Registering ipam driver: "default"
DEBU[0000] releasing IPv4 pools from network bridge (dcfcc71060f02440ae53da5ee0f083ca51c33a290565f1741f451754ae6b4257)
DEBU[0000] ReleaseAddress(LocalDefault/10.254.69.0/24, 10.254.69.1)
DEBU[0000] ReleasePool(LocalDefault/10.254.69.0/24)
DEBU[0000] Allocating IPv4 pools for network bridge (159d0a404ff6564b4fcfe633f0c8c123c0c0606d28ec3b110272650c5fc1bcb6)
DEBU[0000] RequestPool(LocalDefault, 10.254.69.1/24, , map[], false)
DEBU[0000] RequestAddress(LocalDefault/10.254.69.0/24, 10.254.69.1, map[RequestAddressType:com.docker.network.gateway])
DEBU[0000] /sbin/iptables, [--wait -t nat -C POSTROUTING -s 10.254.69.0/24 ! -o docker0 -j MASQUERADE]
DEBU[0000] /sbin/iptables, [--wait -t nat -C DOCKER -i docker0 -j RETURN]
DEBU[0000] /sbin/iptables, [--wait -t nat -I DOCKER -i docker0 -j RETURN]
DEBU[0000] /sbin/iptables, [--wait -D FORWARD -i docker0 -o docker0 -j DROP]
DEBU[0000] /sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT]
DEBU[0000] /sbin/iptables, [--wait -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT]
DEBU[0000] /sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT]
DEBU[0001] /sbin/iptables, [--wait -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0001] /sbin/iptables, [--wait -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0001] /sbin/iptables, [--wait -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8]
DEBU[0001] /sbin/iptables, [--wait -t nat -A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8]
DEBU[0001] /sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -j DOCKER]
DEBU[0001] /sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -j DOCKER]
DEBU[0001] /sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-ISOLATION]
DEBU[0001] /sbin/iptables, [--wait -D FORWARD -j DOCKER-ISOLATION]
DEBU[0001] /sbin/iptables, [--wait -I FORWARD -j DOCKER-ISOLATION]
WARN[0001] Your kernel does not support swap memory limit.
DEBU[0001] Cleaning up old shm/mqueue mounts: start.
DEBU[0001] Cleaning up old shm/mqueue mounts: done.
DEBU[0001] Loaded container 0790b33ec8e5345ac944d560263b8e13cb75f80dd82cd25753c7320bbcb2747c
DEBU[0001] Loaded container 0e36a6c9319e6b7ca4e5b5408e99d77d51b1f4e825248c039ba0260e628c483d
DEBU[0001] Loaded container 135fb2e8cad26d531435dcd19d454e41cf7aece289ddc7374b4c2a984f8b094a
DEBU[0001] Loaded container 2c28de46788ce96026ac8e61e99c145ec55517543e078a781e8ce6c8cddec973
DEBU[0001] Loaded container 35eb075b5815e621378eb8a7ff5ad8652819ec851eaa4f7baedb1383dfa51a57
DEBU[0001] Loaded container 6be37a301a8f52040adf811041c140408224b12599aa55155f8243066d2b0b69
DEBU[0001] Loaded container d98ac7f052fef31761b82ab6c717760428ad5734df4de038d80124ad5b5e8614
DEBU[0001] Starting container 2c28de46788ce96026ac8e61e99c145ec55517543e078a781e8ce6c8cddec973
ERRO[0001] Couldn't run auplink before unmount: exit status 22
ERRO[0001] error locating sandbox id d4c538661db2edc23c79d7dddcf5c7a8886c9477737888a5fc2641bc5e66da8b: sandbox d4c538661db2edc23c79d7dddcf5c7a8886c9477737888a5fc2641bc5e66da8b not found
WARN[0001] failed to cleanup ipc mounts:
failed to umount /var/lib/docker/containers/2c28de46788ce96026ac8e61e99c145ec55517543e078a781e8ce6c8cddec973/shm: invalid argument
ERRO[0001] Failed to start container 2c28de46788ce96026ac8e61e99c145ec55517543e078a781e8ce6c8cddec973: error creating aufs mount to /var/lib/docker/aufs/mnt/187b8026621da2add42330c9393a474fcd9af2e4567596d61bcd7a40c85f71da: invalid argument
INFO[0001] Daemon has completed initialization
INFO[0001] Docker daemon commit=c3959b1 execdriver=native-0.2 graphdriver=aufs version=1.10.2
DEBU[0001] Registering routers
DEBU[0001] Registering HEAD, /containers/{name:.*}/archive
```
and when I try to create new containers by `docker run`, it failed with message:
```
docker: Error response from daemon: error creating aufs mount to /var/lib/docker/aufs/mnt/f9609c0229baa2cdc6bc07c36970ef4f192431c1b1976766b3ea23d72c355df3-init: invalid argument.
See 'docker run --help'.
```
and the daemon log shows:
```
DEBU[0173] Calling POST /v1.22/containers/create
DEBU[0173] POST /v1.22/containers/create
DEBU[0173] form data: {"AttachStderr":false,"AttachStdin":false,"AttachStdout":false,"Cmd":["/hyperkube","kubelet","--api-servers=http://localhost:8080","--v=2","--address=0.0.0.0","--enable-server","--hostname-override=172.16.210.87","--config=/etc/kubernetes/manifests-multi","--cluster-dns=10.253.0.10","--cluster-domain=cluster.local","--allow_privileged=true"],"Domainname":"","Entrypoint":null,"Env":[],"HostConfig":{"Binds":["/sys:/sys:ro","/dev:/dev","/var/lib/docker/:/var/lib/docker:rw","/var/lib/kubelet/:/var/lib/kubelet:rw","/var/run:/var/run:rw","/etc/kubernetes/manifests-multi:/etc/kubernetes/manifests-multi:ro","/:/rootfs:ro"],"BlkioDeviceReadBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceWriteIOps":null,"BlkioWeight":0,"BlkioWeightDevice":null,"CapAdd":null,"CapDrop":null,"CgroupParent":"","ConsoleSize":[0,0],"ContainerIDFile":"","CpuPeriod":0,"CpuQuota":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","Devices":[],"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IpcMode":"","Isolation":"","KernelMemory":0,"Links":null,"LogConfig":{"Config":{},"Type":""},"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NetworkMode":"host","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"host","PidsLimit":0,"PortBindings":{},"Privileged":true,"PublishAllPorts":false,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"always"},"SecurityOpt":null,"ShmSize":0,"UTSMode":"","Ulimits":null,"VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"gcr.io/google_containers/hyperkube:v1.1.8","Labels":{},"NetworkingConfig":{"EndpointsConfig":{}},"OnBuild":null,"OpenStdin":false,"StdinOnce":false,"StopSignal":"SIGTERM","Tty":false,"User":"","Volumes":{},"WorkingDir":""}
ERRO[0173] Couldn't run auplink before unmount: exit status 22
ERRO[0173] Clean up Error! Cannot destroy container 482957f3e4e92a0ba56d4787449daa5a8708f3b77efe0c603605f35d02057566: nosuchcontainer: No such container: 482957f3e4e92a0ba56d4787449daa5a8708f3b77efe0c603605f35d02057566
ERRO[0173] Handler for POST /v1.22/containers/create returned error: error creating aufs mount to /var/lib/docker/aufs/mnt/f9609c0229baa2cdc6bc07c36970ef4f192431c1b1976766b3ea23d72c355df3-init: invalid argument
```
does anyone know whether my approach is correct or not? and why the problem happens after I delete those folders?
I opened #31012 to at least make sure we don't leak these dirs in any circumstances.
We of course also need to look at the various causes of the `busy` errors
This was biting me as long as I can remember. I got pretty much the same results as described above when I switched to `overlay2` driver some days ago and nuked the aufs folder completely (`docker system df` says 1.5Gigs, `df` says 15Gigs).
I had about 1T of diffs using storage. After restarting my docker daemon - I recovered about 700GB. So I guess stopping the daemon prunes these?
Restarting does nothing for me, unfortunately.
Service restart did not help. This is a serious issue. Removing all container/images does not remove those diffs.
Stopping the daemon would not prune these.
If you remove all containers and you still have `diff` dirs, then likely you have some leaked rw layers.
We just encountered this issue. `/var/lib/docker/aufs/diff` took up 28G and took our root filesystem to 100%, which caused our GitLab server to stop responding. We're using docker for GitLab CI. To fix this, I used some of the commands @sogetimaitral suggested above to delete the temp files, and we're back up and running. I restarted the server and sent in a new commit to trigger CI, and everything appears to be working just as it should.
I'm definitely concerned this is going to happen again. What's the deal here? Is this a docker bug that needs to be fixed?
1. Yes there is a bug (both that there are issues on removal and that --force on rm ignores these issues)
2. Generally one should not be writing lots of data to the container fs and instead use a volume (even a throw-away volume). A large diff dir would indicate that there is significant amounts of data being written to the container fs.
If you don't use "--force" on remove you would not run into this issue (or at least you'd see you have a bunch of "dead" containers and know how/what to clean up.).
I'm not manually using docker at all. We're using [gitlab-ci-multi-runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner). Could it be a bug on GitLab's end then?
It looks like (by default) it force-removes containers; https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/dbdbce2848530df299836768c8ea01e209a2fe40/executors/docker/executor_docker.go#L878. Doing so can result in failures to remove the container being ignored, and leading to the orphaned diffs.
Ok, then that tells me that this is a gitlab-ci-multi-runner bug. Is that a correct interpretation? I'm happy to create an issue for them to fix this.
It's a combination I guess; "force" remove makes it easier to handle cleanups (i.e., cases where a container isn't stopped yet, etc), at the same time (that's the "bug" @cpuguy83 mentioned), it can also hide actual issues, such as docker failing to remove the containers filesystem (which can have various reasons). With "force", the container is removed in such cases. Without, the container is left around (but marked "dead")
If the gitlab runner can function correctly without the force remove, that'll probably be good to change (or make it configurable)
I am using [Drone](https://github.com/drone/drone) and have the same issue. I didn't check the code how containers are removed, but i guess it force removes as well.
Could it be a Docker in Docker issue? I am starting Drone with docker-compose.
I decided to go ahead and submit a gitlab-ci-multi-runner issue just to loop the devs in: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/2304
Honestly we worked around this by running Spotify's docker gc with drone CI.
El El mar, mar. 28, 2017 a las 3:38 PM, Geoffrey Fairchild |
notifications@github.com| escribió:
| I decided to go ahead and submit a gitlab-ci-multi-runner issue just to
| loop the devs in:
| https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/2304
|
| —
| You are receiving this because you commented.
| Reply to this email directly, view it on GitHub
| |https://github.com/docker/docker/issues/22207#issuecomment-289926298|,
| or mute the thread
| |https://github.com/notifications/unsubscribe-auth/AC6Wz197zkjWWOlq1-JOibiQP-xJym9Eks5rqYvegaJpZM4IMGt2|
| .
|
@sedouard Thanks for this tip! Running [docker-gc](https://github.com/spotify/docker-gc) hourly from spotify solved the problem for me.
We are getting this issue running from Gitlab CI (not running in docker), using commands to build images / run containers, (not Gitlab CI Docker integration). We are not running any form of force removal, simply `docker run --rm ...` and `docker rmi image:tag`
**EDIT**: sorry, actually the original problem is the same. The difference is that running `spotify/docker-gc` does _not_ fix the problem.
----
As you can see below, I have 0 images, 0 containers, nothing!
`docker system info` agrees with me, but mentions `Dirs: 38` for the aufs storage.
That's suspicious! If you look at `/var/lib/docker/aufs/diff/`, we see that there's actually 1.7 GB of data there, over 41 directories. And that's my personal box, on the production server it's 19 GB.
How do we clean this? using `spotify/docker-gc` does not remove these.
``` shell
philippe@pv-desktop:~$ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
philippe@pv-desktop:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
philippe@pv-desktop:~$ docker system info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 17.03.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 38
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-72-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 31.34 GiB
Name: pv-desktop
ID: 2U5D:CRHS:RUQK:YSJX:ZTRS:HYMV:HO6Q:FDKE:R6PK:HMUN:2EOI:RUWO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: silex
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
philippe@pv-desktop:~$ ls -alh /var/lib/docker/aufs/diff/
total 276K
drwxr-xr-x 40 root root 116K Apr 13 15:32 .
drwxr-xr-x 5 root root 4.0K Sep 18 2015 ..
drwxr-xr-x 4 root root 4.0K Jun 17 2016 005d00efb0ba949d627ad439aec8c268b5d55759f6e92e51d7828c12e3817147
drwxr-xr-x 8 root root 4.0K May 2 2016 0968e52874bbfaa938ffc869cef1c5b78e2d4f7a670e19ef47f713868b9bfbdf
drwxr-xr-x 4 root root 4.0K Jun 20 2016 188233e6dcc37e2308e69807ffd19aca3e61be367daae921f2bcb15a1d6237d0
drwxr-xr-x 6 root root 4.0K Jun 20 2016 188233e6dcc37e2308e69807ffd19aca3e61be367daae921f2bcb15a1d6237d0-init
drwxr-xr-x 21 root root 4.0K Apr 8 2016 250ecb97108a6d8a8c41f9d2eb61389a228c95f980575e95ee61f9e8629d5180
drwxr-xr-x 2 root root 4.0K Dec 22 2015 291f16f99d9b0bc05100e463dbc007ef816e0cf17b85d20cf51da5eb2b866810
drwxr-xr-x 2 root root 4.0K May 2 2016 3054baaa0b4a7b52da2d25170e9ce4865967f899bdf6d444b571e57be141b712
drwxr-xr-x 2 root root 4.0K Feb 5 2016 369aca82a5c05d17006b9dca3bf92d1de7d39d7cd908ed665ef181649525464e
drwxr-xr-x 3 root root 4.0K Jun 17 2016 3835a1d1dfe755d9d1ada6933a0ea7a4943caf8f3d96eb3d79c8de7ce25954d2
(...strip)
philippe@pv-desktop:~$ du -hs /var/lib/docker/aufs/diff/
1.7G /var/lib/docker/aufs/diff/
philippe@pv-desktop:~$ docker system prune -a
WARNING! This will remove:
- all stopped containers
- all volumes not used by at least one container
- all networks not used by at least one container
- all images without at least one container associated to them
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0 B
```
Can I safely `rm -r /var/lib/docker/aufs` and restart the docker deamon?
Running `spotify/docker-gc` does not clean those orphans.
**EDIT**: thanks @CVTJNII!
Stopping the Docker daemon and erasing all of /var/lib/docker will be safer. Erasing /var/lib/docker/aufs will cause you to lose your images anyway so it's better to start with a clean /var/lib/docker in my opinion. This is the "solution" I've been using for several months for this problem now.
Starting with 17.06 there should no longer be any *new* orphaned diffs.
Instead you may start seeing containers with the state `Dead`, this happens if there was an error during removal that is non-recoverable and may require an admin to deal with it.
In addition, removal is a bit more robust, and less prone to error due to race conditions or failed unmounts.
@cpuguy83: great news, can you explain what the admin would need to do if that happens?
@Silex It depends on the cause.
Typically what has happened is there is a `device or resource busy` error due to some mount being leaked into a container. If you are running something like cadvisor this is pretty much a guarantee as the instructions say to mount the whole docker dir into the cadvisor container.
This *can* be tricky, you may have to stop the offending container(s) and then remove the `dead` container.
If you are on a newer kernel (3.15+) it is unlikely that you would see the issue anymore, though there still may be some edge case.
Docker version 17.06.0-ce, build 02c1d87
I tried remove all images, volumes, networks and containers but it not helped.
Also tried commands:
```
docker system prune -af
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc:ro spotify/docker-gc
```
Still remain files:
```
root@Dark:/var/lib/docker/aufs# ls -la *
diff:
total 92
drwx------ 12 root root 45056 Jul 28 17:28 .
drwx------ 5 root root 4096 Jul 9 00:18 ..
drwxr-xr-x 4 root root 4096 Jul 10 01:35 78f8ecad2e94fedfb1ced425885fd80bb8721f9fd70715de2ce373986785b882
drwxr-xr-x 6 root root 4096 Jul 10 01:35 78f8ecad2e94fedfb1ced425885fd80bb8721f9fd70715de2ce373986785b882-init
drwxr-xr-x 5 root root 4096 Jul 10 01:35 7caa9688638ea9669bac451b155b65b121e99fcea8d675688f0c76678ba85ccd
drwxr-xr-x 6 root root 4096 Jul 10 01:35 7caa9688638ea9669bac451b155b65b121e99fcea8d675688f0c76678ba85ccd-init
drwxr-xr-x 4 root root 4096 Jul 12 14:45 b7b7770aae461af083e72e5e3232a62a90f934c83e38830d06365108e302e7ac
drwxr-xr-x 6 root root 4096 Jul 12 14:45 b7b7770aae461af083e72e5e3232a62a90f934c83e38830d06365108e302e7ac-init
drwxr-xr-x 4 root root 4096 Jul 10 01:35 d5752b27b341e17e730d3f4acbec04b10e41dc01ce6f9f98ff38208c0647f2e4
drwxr-xr-x 6 root root 4096 Jul 10 01:35 d5752b27b341e17e730d3f4acbec04b10e41dc01ce6f9f98ff38208c0647f2e4-init
drwxr-xr-x 6 root root 4096 Jul 10 01:35 e412d3c6f0f5f85e23d7a396d47c459f5d74378b474b27106ab9b82ea829dbfb
drwxr-xr-x 6 root root 4096 Jul 10 01:35 e412d3c6f0f5f85e23d7a396d47c459f5d74378b474b27106ab9b82ea829dbfb-init
layers:
total 52
drwx------ 2 root root 45056 Jul 28 17:28 .
drwx------ 5 root root 4096 Jul 9 00:18 ..
-rw-r--r-- 1 root root 0 Jul 10 01:35 78f8ecad2e94fedfb1ced425885fd80bb8721f9fd70715de2ce373986785b882
-rw-r--r-- 1 root root 0 Jul 10 01:35 78f8ecad2e94fedfb1ced425885fd80bb8721f9fd70715de2ce373986785b882-init
-rw-r--r-- 1 root root 0 Jul 10 01:35 7caa9688638ea9669bac451b155b65b121e99fcea8d675688f0c76678ba85ccd
-rw-r--r-- 1 root root 0 Jul 10 01:35 7caa9688638ea9669bac451b155b65b121e99fcea8d675688f0c76678ba85ccd-init
-rw-r--r-- 1 root root 0 Jul 12 14:45 b7b7770aae461af083e72e5e3232a62a90f934c83e38830d06365108e302e7ac
-rw-r--r-- 1 root root 0 Jul 12 14:45 b7b7770aae461af083e72e5e3232a62a90f934c83e38830d06365108e302e7ac-init
-rw-r--r-- 1 root root 0 Jul 10 01:35 d5752b27b341e17e730d3f4acbec04b10e41dc01ce6f9f98ff38208c0647f2e4
-rw-r--r-- 1 root root 0 Jul 10 01:35 d5752b27b341e17e730d3f4acbec04b10e41dc01ce6f9f98ff38208c0647f2e4-init
-rw-r--r-- 1 root root 0 Jul 10 01:35 e412d3c6f0f5f85e23d7a396d47c459f5d74378b474b27106ab9b82ea829dbfb
-rw-r--r-- 1 root root 0 Jul 10 01:35 e412d3c6f0f5f85e23d7a396d47c459f5d74378b474b27106ab9b82ea829dbfb-init
mnt:
total 92
drwx------ 12 root root 45056 Jul 28 17:28 .
drwx------ 5 root root 4096 Jul 9 00:18 ..
drwxr-xr-x 2 root root 4096 Jul 10 01:35 78f8ecad2e94fedfb1ced425885fd80bb8721f9fd70715de2ce373986785b882
drwxr-xr-x 2 root root 4096 Jul 10 01:35 78f8ecad2e94fedfb1ced425885fd80bb8721f9fd70715de2ce373986785b882-init
drwxr-xr-x 2 root root 4096 Jul 10 01:35 7caa9688638ea9669bac451b155b65b121e99fcea8d675688f0c76678ba85ccd
drwxr-xr-x 2 root root 4096 Jul 10 01:35 7caa9688638ea9669bac451b155b65b121e99fcea8d675688f0c76678ba85ccd-init
drwxr-xr-x 2 root root 4096 Jul 12 14:45 b7b7770aae461af083e72e5e3232a62a90f934c83e38830d06365108e302e7ac
drwxr-xr-x 2 root root 4096 Jul 12 14:45 b7b7770aae461af083e72e5e3232a62a90f934c83e38830d06365108e302e7ac-init
drwxr-xr-x 2 root root 4096 Jul 10 01:35 d5752b27b341e17e730d3f4acbec04b10e41dc01ce6f9f98ff38208c0647f2e4
drwxr-xr-x 2 root root 4096 Jul 10 01:35 d5752b27b341e17e730d3f4acbec04b10e41dc01ce6f9f98ff38208c0647f2e4-init
drwxr-xr-x 2 root root 4096 Jul 10 01:35 e412d3c6f0f5f85e23d7a396d47c459f5d74378b474b27106ab9b82ea829dbfb
drwxr-xr-x 2 root root 4096 Jul 10 01:35 e412d3c6f0f5f85e23d7a396d47c459f5d74378b474b27106ab9b82ea829dbfb-init
```
```
# docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 0 0 0B 0B
Containers 0 0 0B 0B
Local Volumes 0 0 0B 0B
```
How can it be deleted?
@haos616 try stopping all running containers first, and then run `docker system prune -af`.
This did the trick for me.
Didn't work while I had a container running.
If it's an upgrade from a previous version of docker, it's possible those diffs were generated / left behind by that version. Docker 17.06 won't remove a container if layers failed to be removed (when using --force); older versions did, which could lead to orphaned layers
@julian-pani I did so in the beginning but it does not help.
```
# docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 0 0 0B 0B
Containers 0 0 0B 0B
Local Volumes 0 0 0B 0B
```
@thaJeztah No. I cleaned the Docker one or two months ago. Then the version was already 17.06. I used command `docker system prune -af`. It removed everything.
Running https://github.com/spotify/docker-gc as a container worked for me, but it went a step extra and deleted some of my required images too :(
So I've put a small wrapper script as below to be safe
```
#!/bin/sh
docker images -q | /etc/docker-gc-exclude # Save all genuine images as exclude
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc:ro spotify/docker-gc
```
thanks again to spotify
IIUC, the spotify script just calls `docker rm` and `docker rmi` - did it actually remove orphaned diffs?
Just some feedback for the community, I've read through all of this and none of the solutions actually seem to work consistently or reliably. My "fix" was simply to double the amount of disc space on my AWS instances. And I know all too well that's a crappy fix but it is the best workaround I've found to Docker's bloated aufs. This really, really needs to be fixed.
@fuzzygroup 17.06 should no longer create orphaned diffs, but it won't clean up the old ones yet.
I could cleanup with this script. I don't see why it wouldn't work, but who knows.
Anyway it's working fine for me. It will delete all images, containers, and volumes... As it should not run very often, I find it a minor side effect. But it's up to you to use it or not.
https://gist.github.com/Karreg/84206b9711cbc6d0fbbe77a57f705979
https://stackoverflow.com/q/45798076/562769 seems to be related. I've posted a quick fix.
FYI, still seeing this with `17.06.1-ce`
```
Containers: 20
Running: 0
Paused: 0
Stopped: 20
Images: 124
Server Version: 17.06.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 185
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
apparmor
Kernel Version: 4.4.0-83-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 7.796GiB
Name: gitlab-cirunner
ID: PWLR:R6HF:MK3Y:KN5A:AWRV:KHFY:F36D:WASF:7K7B:U7FY:2DJA:DBE2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
```
`/var/lib/docker/aufs/diff` contains lots of directories with the `-init-removing` and `-removing` prefix:
```
ffd5477de24b0d9993724e40175185038a62250861516030a33280898243e742-init-removing
ffd5477de24b0d9993724e40175185038a62250861516030a33280898243e742-removing
ffd900de0634992e99c022a16775805dfd0ffd1f6c89fece7deb6b1a71c5e38c-init-removing
ffd900de0634992e99c022a16775805dfd0ffd1f6c89fece7deb6b1a71c5e38c-removing
```
| FYI, still seeing this with 17.06.1-ce
Still seeing what, exactly?
There should not be any way that a diff dir can leak, though diff dirs will still exist if they existed on upgrade, they'll still exist.
Still seeing orphaned diffs as far as I can tell. `docker system prune` didn't remove them, neither did `docker-gc`. Manually running `rm -rf /var/lib/docker/aufs/diff/*-removing` seems to be working.
Yes, docker will not clean up old orphaned dirs yet.
By old you mean those created from a previous version of docker with this issue?
This is a fresh install of Docker we did about two weeks ago, those orphans must have been created since then, so it seems that docker must still be creating those orphans?
I mean, in the last half an hour I've got `112` new diffs with `-removing`, since I rm'ed them manually.
```
$ ls /var/lib/docker/aufs/diff/ | grep removing | wc -l
112
```
You said "17.06 should no longer create orphaned diffs, but it won't clean up the old ones yet.", but surely this cannot be correct, or am I missing something? Are those tagged with `-removing` not orphaned?
@orf On a newer kernel, it's highly unexpected to have any issue at all during removal. Are you mounting `/var/lib/docker` into a container?
I'll check in the aufs driver to see if there's a specific issue there with it reporting a successful remove when it really wasn't.
We are not mounting `/var/lib/docker` into a container.
```
$ uname -a
Linux gitlab-cirunner 4.4.0-83-generic #106~14.04.1-Ubuntu SMP Mon Jun 26 18:10:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
```
We are running 14.04 LTS
Let me know if there is anything I can provide to help debug this.
For other reasons (swarm mode networking) I moved off 14.04 for Docker
machines.
On Mon, Aug 21, 2017 at 8:23 AM orf |notifications@github.com| wrote:
| We are not mounting /var/lib/docker into a container.
|
| $ uname -a
| Linux gitlab-cirunner 4.4.0-83-generic #106~14.04.1-Ubuntu SMP Mon Jun 26 18:10:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
|
| We are running 14.04 LTS
|
| —
| You are receiving this because you commented.
| Reply to this email directly, view it on GitHub
| |https://github.com/moby/moby/issues/22207#issuecomment-323773033|, or mute
| the thread
| |https://github.com/notifications/unsubscribe-auth/AADRIfE2B2HNpbsKyTOj1CwGzulRT2C0ks5saaDPgaJpZM4IMGt2|
| .
|
This appears to be worse with 17.06.01-ce. I updated a build machine to this version and immediately started seeing the `*-init-removing` and the `*-removing` directories left around as part of the build process. I stopped the service, removed the `/var/lib/docker` directory, restarted the service and after a few builds was close to out of disk space again. I stopped the service again, ran `apt-get purge docker-ce`, removed `/var/lib/docker` again and installed the 17.06.0-ce version. Not getting the extra directories in `/var/lib/docker/aufs/diff` and disk space is representative of images that are on the build machine. I've reproduced the behavior on my development machine as well - just building an image seems to create these extra directories for each layer of the image so I would run out of disk space really quick. Again, reverting to 17.06.0-ce seems to not have the problem so I'm going to stay there for now.
@mmanderson Thanks for reporting. Taking a look at changes in the AUFS driver.
@mmanderson Do you have any containers in the `Dead` state in `docker ps -a`?
All of my docker build servers are running out of space.
I have upgraded within the last week or so to Docker version 17.06.1-ce, build 874a737. I believe that nothing else has changed and that this issue either emerged or manifested as part of the upgrade process. The aufs diff directory is massive and I already pruned all images and dangling volumes.
[issue-22207.txt](https://github.com/moby/moby/files/1240116/issue-22207.txt)
@cpuguy83 No containers in any state. Here is what I just barely did to demonstrate this with 17.06.01-ce:
1. Started with a fresh install of docker 17.06.01-ce on Ubuntu 16.04.03 LTS (i.e. docker not installed and no /var/lib/docker directory). After install verified an empty /var/lib/docker/aufs/diff directory.
2. Ran a docker build with a fairly simple dockerfile based on ubuntu:latest - all it does is pull statsd_exporter from github and extract it into /usr/bin (see attached file).
3. After running the build run `docker ps -a` to show no containers in any state. There are several `*-remaining` folder in the `/var/lib/docker/aufs/diff` folder.
4. Run `docker system df` to verify images, container, and volumes. Result is
```
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 2 0 132.7MB 132.7MB (100%)
Containers 0 0 0B 0B
Local Volumes 0 0 0B 0B
```
6. Running `du -sch /var/lib/docker/*/` shows 152M for `/var/lib/docker/aufs/`
7. Run `docker rmi $(docker images -q)` to remove the built image layers. Running `docker system df` after this shows all zeros. Running `du -sch /var/lib/docker/*/` shows 152M for `/var/lib/docker/aufs/` and there are `*-remaining` folders for all of the folders that didn't have them before along with the existing `*-remaining` folders that are still there.
@erikh is this the issue you are experiencing?
@cpuguy83 After uninstalling 17.06.01-ce, removing the /var/lib/docker directory and installing 17.06.0-ce I try to run the same build. The build fails because of the `ADD from remote URL's` bug that was fixed in 17.06.01. However I don't get any `*-remaining` directories for the steps that do complete and after cleaning up everything with `docker system prune` and `docker rmi $(docker image -q)` the `/var/lib/docker/aufs/diff` directory is again empty and the space is freed.
Thanks all, this is a regression in 17.06.1...
PR to fix is here: https://github.com/moby/moby/pull/34587
awesome, thanks for the quick patch @cpuguy83! /cc @erikh
@rogaha! yes, thanks to you and @cpuguy83!
Thank you so much @Karreg for your [excellent script](https://github.com/moby/moby/issues/22207#issuecomment-322707352). After getting rid of all the old ophaned diffs and freeing huge amounts of lost disk space again we are using it now regularily to clean our VMs before installing new docker images. Great help and an almost perfect workaround for this issue now. @TP75
Looks like Docker, Inc. have some contracts with computer data storage manufacturers.
@Karreg's script worked fine for me and I freed all the space in the diffs directory.
Having the same issue.
Docker Host Details
root@UbuntuCont:~# docker info
Containers: 3
Running: 0
Paused: 0
Stopped: 3
Images: 4
Server Version: 17.06.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 14
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-93-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.358GiB
Name: UbuntuCont
ID: QQA5:DC5S:C2FL:LCC6:XY6E:V3FR:TRW3:VMOQ:QQKD:AP2M:H3JA:I6VX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
root@UbuntuCont:/var/lib/docker/aufs/diff# ls
031c85352fe85f07fede77dee0ac9dc2c7723177a819e72c534e1399208c95fa
09d53040e7e6798b5987ea76fe4f84f0906785b94a392a72e8e41a66cd9f242d
09d53040e7e6798b5987ea76fe4f84f0906785b94a392a72e8e41a66cd9f242d-init
0fb1ffc90969e9706801e2a18870f3ecd857a58f1094fbb968b3fa873e4cf2e4
10549179bd21a9c7af018d4ef305bb9196413b9662fce333b607104c40f38781
10d86a48e03cabf9af2c765dc84824809f24674ac339e4b9ffe572f50bd26b9c-init-removing
10d86a48e03cabf9af2c765dc84824809f24674ac339e4b9ffe572f50bd26b9c-removing
2e226946e8e6c2b3613de2afcff4cbb9890b6d9bd365fdda121a51ae96ec5606
2e226946e8e6c2b3613de2afcff4cbb9890b6d9bd365fdda121a51ae96ec5606-init
3601f6953132f557df8b52e03016db406168d3d6511d7ff5c08a90925ea288da-init-removing
3601f6953132f557df8b52e03016db406168d3d6511d7ff5c08a90925ea288da-removing
4b29141243aea4e70472f25a34a91267ab19c15071862c53e903b99740603d4c-init-removing
4b29141243aea4e70472f25a34a91267ab19c15071862c53e903b99740603d4c-removing
520e3fcf82e0fbbb48236dd99b6dee4c5bb9073d768511040c414f205c787dc5-init-removing
520e3fcf82e0fbbb48236dd99b6dee4c5bb9073d768511040c414f205c787dc5-removing
59cbb25a4858e7d3eb9146d64ff7602c9abc68509b8f2ccfe3be76681481904f
5d1c661b452efce22fe4e109fad7a672e755c64f538375fda21c23d49e2590f6
605893aba54feee92830d56b6ef1105a4d2166e71bd3b73a584b2afc83319591
63bd53412210f492d72999f9263a290dfee18310aa0494cb92e0d926d423e281-init-removing
63bd53412210f492d72999f9263a290dfee18310aa0494cb92e0d926d423e281-removing
72146e759ab65c835e214e99a2037f4b475902fdbe550c46ea0d396fb5ab2779-init-removing
72146e759ab65c835e214e99a2037f4b475902fdbe550c46ea0d396fb5ab2779-removing
8147e0b06dcbce4aa7eb86ed74f4ee8301e5fe2ee73c3a80dcb230bd0ddfcc26-init-removing
8147e0b06dcbce4aa7eb86ed74f4ee8301e5fe2ee73c3a80dcb230bd0ddfcc26-removing
a72735551217bb1ad01b77dbdbb9b8effa9f41315b0c481f8d74b5606c50deb4
aa58f2000b9f7d1ed2a6b476740c292c3c716e1d4dc04b7718580a490bba5ee8
b552cb853e33a8c758cb664aec70e2c4e85eacff180f56cbfab988a8e10c0174-removing
cd80c351b81ed13c4b64d9dfdc20c84f6b01cbb3e26f560faf2b63dae12dec55-init-removing
cd80c351b81ed13c4b64d9dfdc20c84f6b01cbb3e26f560faf2b63dae12dec55-removing
fe903be376821b7afee38a016f9765136ecb096c59178156299acb9f629061a2
fe903be376821b7afee38a016f9765136ecb096c59178156299acb9f629061a2-init
@kasunsjc please read the posts just above yours.
I confirm upgrading to 17.06.2-ce solved this issue. I didn't have to manually the directories either (last time) after the upgrade.
17.06.2-ce _appears_ to have fixed this for me as well. No more `-removing` directories in there, got a decent amount of space back.
I'm assuming that the `-init` directories I have in `aufs/diff` are unrelated (some of them are pretty old). They are all small, though, so it hardly matters.
Updating to 17.07.0 solved the issue here too, not even `docker system prune --all -f` would remove the directories before but after upgrading they got autoremoved on reboot.
Confirming this issue was resolved on Ubuntu 16.04 with 17.06.2-ce. As soon as it was updated, the space cleared.
True class: unrelated
Text with highlighted words
I'd like to know why docker uses so much disk, even after removing _all_ containers, images, and volumes.
It looks like this "diff" has a layer, but the layer isn't referenced by anything at all.
```
/var/lib/docker/aufs/diff# du-summary
806628 c245c4c6d71ecdd834974e1e679506d33c4aac5f552cb4b28e727a596efc1695-removing
302312 a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
302304 957e78f9f9f4036689734df16dabccb98973e2c3de0863ef3f84de85dca8d92d
302256 8db1d610f3fbc71415f534a5d88318bbd2f3f783375813f2288d15f15846d312
288204 ac6b8ff4c0e7b91230ebf4c1caf16f06c1fdceff6111fd58f4ea50fc2dd5050b
288180 04a478c413ea80bcfa7f6560763beef991696eace2624254479e5e5dd69708c6
287804 d033ab6e4e5231dc46c6c417c680b239bb0e843024738517cbb0397128e166ca
233420 8e21143dca49e30cae7475b71b5aee9b92abe2069fbb9ab98ce9c334e3f6d4fa
212668 a631b94f7a2d5d21a96a78e9574d39cdeebbc81b51ac6c58bd48dc4045656477
205120 ae13341f8c08a925a95e5306ac039b0e0bbf000dda1a60afb3d15c838e43e349
205120 8d42279017d6095bab8d533ab0f1f7de229aa7483370ef53ead71fe5be3f1284
205116 59b3acd8e0cfd194d44313978d4b3769905cdb5204a590069c665423b10150e3
205116 040af0eee742ec9fb2dbeb32446ce44829cd72f02a2cf31283fcd067e73798ab
158024 ef0a29ff0b515c8c57fe78bcbd597243de9f7b274d9b212c774d91bd45a6c9b1
114588 061bd7e021afd4aaffa9fe6a6de491e10d8d37d9cbe7612138f58543e0985280
114576 149e8d2745f6684bc2106218711991449c452d4c7e6203e2a0f46651399162b0
114532 52b28112913abb0ed1b3267a0baa1cacd022ca6611812d0a8a428e61ec399589
114300 52475beba19687a886cba4bdb8508d5aaf051ceb52fb3a65294141ab846c8294
76668 4e6afb958b5ee6dea6d1a886d19fc9c780d4ecc4baeebfbde31f9bb97732d10d
76640 c61340c6a962ddd484512651046a676dbbc6a5d46aecc26995c49fe987bf9cdc
/var/lib/docker/aufs/diff# du -hs a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
296M a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
$ docker-find a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
+ docker=/var/lib/docker
+ sudo find /var/lib/docker '(' -path '/var/lib/docker/aufs/diff/*' -o -path '/var/lib/docker/aufs/mnt/*' ')' -prune -o -print
+ grep a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
/var/lib/docker/aufs/layers/a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
+ sudo find /var/lib/docker '(' -path '/var/lib/docker/aufs/diff/*' -o -path '/var/lib/docker/aufs/mnt/*' ')' -prune -o -type f -print0
+ sudo xargs -0 -P20 grep -l a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
/var/lib/docker/aufs/layers/993e4988c510ec3ab4f6d139740a059df40585576f8196817e573a9684554c5c
/var/lib/docker/aufs/layers/95e68d59a8704f2bb52cc1306ca910ddb7af8956eb7c57970fcf7d8b3d9baddb
/var/lib/docker/aufs/layers/4e6afb958b5ee6dea6d1a886d19fc9c780d4ecc4baeebfbde31f9bb97732d10d
/var/lib/docker/aufs/layers/fd895b6f56aedf09c48dba97931a34cea863a21175450c31b6ceadde03f7b3da
/var/lib/docker/aufs/layers/ac6b8ff4c0e7b91230ebf4c1caf16f06c1fdceff6111fd58f4ea50fc2dd5050b
/var/lib/docker/aufs/layers/f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579-init
/var/lib/docker/aufs/layers/d5bbef5adf2efb6f15d4f96c4bee21beb955255d1ec17baf35de66e98e6c7328
/var/lib/docker/aufs/layers/9646360df378b88eae6f1d6288439eebd9647d5b9e8a471840d4a9d6ed5d92a4
/var/lib/docker/aufs/layers/cf9fd1c4a64baa39b6d6d9dac048ad2fff3c3fe13924b07377e767eed230ba9f
/var/lib/docker/aufs/layers/f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579
/var/lib/docker/aufs/layers/23ce5a473b101d85f0e9465debe5a0f3b8a2079b99528a797b02052d06bc11d8
/var/lib/docker/image/aufs/layerdb/sha256/d1c659b8e3d0e893e95c8eedc755adcb91a1c2022e1090376b451f7206f9b1c0/cache-id
$ sudo cat /var/lib/docker/image/aufs/layerdb/sha256/d1c659b8e3d0e893e95c8eedc755adcb91a1c2022e1090376b451f7206f9b1c0/diff
sha256:b5185949ba02a6e065079660b0536672c9691fb0e0cb1fd912b2c7b29c91d625
$ docker-find sha256:b5185949ba02a6e065079660b0536672c9691fb0e0cb1fd912b2c7b29c91d625
+ docker=/var/lib/docker
+ sudo find /var/lib/docker '(' -path '/var/lib/docker/aufs/diff/*' -o -path '/var/lib/docker/aufs/mnt/*' ')' -prune -o -print
+ grep sha256:b5185949ba02a6e065079660b0536672c9691fb0e0cb1fd912b2c7b29c91d625
+ sudo find /var/lib/docker '(' -path '/var/lib/docker/aufs/diff/*' -o -path '/var/lib/docker/aufs/mnt/*' ')' -prune -o -type f -print0
+ sudo xargs -0 -P20 grep -l sha256:b5185949ba02a6e065079660b0536672c9691fb0e0cb1fd912b2c7b29c91d625
/var/lib/docker/image/aufs/layerdb/sha256/d1c659b8e3d0e893e95c8eedc755adcb91a1c2022e1090376b451f7206f9b1c0/diff
```
+```
# docker --version
Docker version 1.10.3, build 99b71ce
# docker info
Containers: 3
Running: 0
Paused: 0
Stopped: 3
Images: 29
Server Version: 1.10.3
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 99
Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 3.13.0-83-generic
Operating System: |unknown|
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 125.9 GiB
Name: dev34-devc
ID: VKMX:YMJ2:3NGV:5J6I:5RYM:AVBK:QPOZ:ODYE:VQ2D:AF2J:2LEM:TKTE
WARNING: No swap limit support
```
I should also show that docker lists no containers, volumes, or images:
```
$ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ docker volume ls
DRIVER VOLUME NAME
```
strange; especially because of;
```
Containers: 3
Running: 0
Paused: 0
Stopped: 3
Images: 29
```
which doesn't match the output of `docker images` / `docker ps`.
What operating system are you running on?
```
Operating System: |unknown|
```
@tonistiigi any idea?
That was afterward. I guess some processes kicked off in the meantime.
The state I'm referring to (I have now) is:
```
$ docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
```
And I still have:
```
$ sudo du -hs /var/lib/docker/aufs/diff/a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
296M /var/lib/docker/aufs/diff/a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
```
We're on Ubuntu Lucid with an upgraded kernel =/
```
$ uname -a
Linux dev34-devc 3.13.0-83-generic #127-Ubuntu SMP Fri Mar 11 00:25:37 UTC 2016 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 10.04.1 LTS
Release: 10.04
Codename: lucid
```
It seems an interesting issue.
is it possible to have way to reproduce it ? @bukzor
Surely it's possible, but I don't know how.
Please try running the below script on one of your active docker hosts and see what's left.
In our case, there's always plenty of diffs left behind.
``` #!bash
#!/bin/bash
set -eu
echo "WARNING:: This will stop ALL docker processes and remove ALL docker images."
read -p "Continue (y/n)? "
if [ "$REPLY" != "y" ]; then
echo "Aborting."
exit 1
fi
xdocker() { exec xargs -P10 -r -n1 --verbose docker "$@"; }
set -x
# remove containers
docker ps -q | xdocker stop
docker ps -aq | xdocker rm
# remove tags
docker images | sed 1d | grep -v '^|none|' | col 1 2 | sed 's/ /:/' | xdocker rmi
# remove images
docker images -q | xdocker rmi
docker images -aq | xdocker rmi
# remove volumes
docker volume ls -q | xdocker volume rm
```
One possible way I see this happening is that if there are errors on aufs unmounting. For example, if there are EBUSY errors then probably the image configuration has already been deleted before.
@bukzor Would be very interesting if there was a reproducer that would start from an empty graph directory, pull/run images and get it into a state where it doesn't fully clean up after running your script.
That would be interesting, but sounds like a full day's work.
I can't commit to that.
Here's some more data regarding the (arbitrarily selected) troublesome diff above, `a800`.
``` #!sh
$ docker-find a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea | sudo xargs -n1 wc -l | sort -rn
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -print
+ grep a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -type f -print0
+ sudo xargs -0 -P20 grep -l a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
15 /nail/var/lib/docker/aufs/layers/f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579
14 /nail/var/lib/docker/aufs/layers/f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579-init
13 /nail/var/lib/docker/aufs/layers/993e4988c510ec3ab4f6d139740a059df40585576f8196817e573a9684554c5c
12 /nail/var/lib/docker/aufs/layers/cf9fd1c4a64baa39b6d6d9dac048ad2fff3c3fe13924b07377e767eed230ba9f
11 /nail/var/lib/docker/aufs/layers/4e6afb958b5ee6dea6d1a886d19fc9c780d4ecc4baeebfbde31f9bb97732d10d
10 /nail/var/lib/docker/aufs/layers/23ce5a473b101d85f0e9465debe5a0f3b8a2079b99528a797b02052d06bc11d8
9 /nail/var/lib/docker/aufs/layers/95e68d59a8704f2bb52cc1306ca910ddb7af8956eb7c57970fcf7d8b3d9baddb
8 /nail/var/lib/docker/aufs/layers/ac6b8ff4c0e7b91230ebf4c1caf16f06c1fdceff6111fd58f4ea50fc2dd5050b
7 /nail/var/lib/docker/aufs/layers/fd895b6f56aedf09c48dba97931a34cea863a21175450c31b6ceadde03f7b3da
6 /nail/var/lib/docker/aufs/layers/d5bbef5adf2efb6f15d4f96c4bee21beb955255d1ec17baf35de66e98e6c7328
5 /nail/var/lib/docker/aufs/layers/9646360df378b88eae6f1d6288439eebd9647d5b9e8a471840d4a9d6ed5d92a4
4 /nail/var/lib/docker/aufs/layers/a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
0 /nail/var/lib/docker/image/aufs/layerdb/sha256/d1c659b8e3d0e893e95c8eedc755adcb91a1c2022e1090376b451f7206f9b1c0/cache-id
```
So we see there's a chain of child layers, with `f3286009193` as the tip.
```
$ docker-find f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579'$'
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -print
+ grep --color 'f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579$'
/nail/var/lib/docker/aufs/layers/f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -type f -print0
+ sudo xargs -0 -P20 grep --color -l 'f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579$'
/nail/var/lib/docker/image/aufs/layerdb/mounts/eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e/mount-id
```
So that layer was used in mount `eb809c0321`. I don't find any references to that mount anywhere:
```
$ docker-find eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -print
+ grep --color eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e
/nail/var/lib/docker/image/aufs/layerdb/mounts/eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e
/nail/var/lib/docker/image/aufs/layerdb/mounts/eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e/mount-id
/nail/var/lib/docker/image/aufs/layerdb/mounts/eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e/init-id
/nail/var/lib/docker/image/aufs/layerdb/mounts/eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e/parent
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -type f -print0
+ sudo xargs -0 -P20 grep --color -l eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e
```
Is there any way to find what container that mount was used for?
The doc only says the mount ID is no longer equal to the container ID, which isn't very helpful.
https://docs.docker.com/engine/userguide/storagedriver/aufs-driver/
@bukzor `eb809c0321` is the container id. What docs mean is that aufs id (`f3286009193f` in your case) is not container id.
/cc @dmcgowan as well
@tonistiigi OK.
Then obviously the mount has outlived its container.
At what point in the container lifecycle is the mount cleaned up?
Is this the temporary writable aufs for running/stopped containers?
@bukzor (rw) mount is deleted on container deletion. Unmount happens on container process stop. Diff folders are a place where the individual layer contents are stored, it doesn't matter is the layer is mounted or not.
@bukzor The link between the aufs id and container id can be found at `image/aufs/layerdb/mounts/|container-id|/mount-id`. From just knowing an aufs id the easiest way to find the container id is to grep the `image/aufs/layerdb` directory for it. If nothing is found, then the cleanup was not completed cleanly.
Running into similar issue.
We're running daily CI in the docker daemon server. /var/lib/docker/aufs/diff takes quite a mount of disk capacity, which it shouldn't be.
Still `2gb` in `aufs/diff` after trying everything reasonable suggested here or in related threads (including @bukzor's bash script above).
Short of a proper fix, is there any straightforward way of removing the leftover mounts without removing all other images at the same time? (If no containers are running currently, I guess there should be no mounts, right?)
I am experiencing the same issue. I am using this machine to test a lot of containers, then commit/delete. My /var/lib/docker/aufs directory is currently 7.9G heavy. I'm going to have to move this directory to another mount point, because storage on this one is limited. :(
```
# du -sh /var/lib/docker/aufs/diff/
1.9T /var/lib/docker/aufs/diff/
```
@mcallaway Everything in `aufs/diff` is going to be fs writes performed in a container.
I have the same issue. All containers which I have are in running state, but there are lots of aufs diff directories which don't relate to these containers and relate to old removed containers. I can remove them manually, but it is not an option. There should be a reason for such a behavior.
I use k8s 1.3.5 and docker 1.12.
Running of the `docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc spotify/docker-gc` helped.
I have the same issue. I'm using Gitlab CI with dind (docker in docker).
IMHO when image in the registry was updated within the same tag and it was pulled, then related container was restarted, old container and image are not GCed unless you run `spotify/docker-gc`.
Can someone else confirm this?
@kayrus correct, docker will not automatically assume that an "untagged" image should also be _removed_. Containers could still be using that image, and you can still start new containers from that image (referencing it by its ID). You can remove "dangling" images using `docker rmi $(docker images -qa -f dangling=true)`. Also, docker 1.13 will get data management commands (see https://github.com/docker/docker/pull/26108), which allow you to more easily cleanup unused images, containers, etc.
@thaJeztah does `/var/lib/docker/aufs/diff/` actually contain the "untagged" images?
@kayrus yes they are part of the images (tagged, and untagged)
getting a similar issue, no containers/images/volumes, ~13Gb of diffs
```
$ docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 1030
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runcSecurity Options: apparmor
Kernel Version: 3.13.0-32-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.861 GiB
Name: gitrunner
ID: GSAW:6X5Z:SHHU:NZIM:O76D:P5OE:7OZG:UFGQ:BOAJ:HJFM:5G6W:5APP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): falseRegistry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
```
```
$ docker volume ls
DRIVER VOLUME NAME
$ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$
```
```
$ df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/mapper/gitrunner--docker-lib--docker 18G 15G 2.6G 85% /var/lib/docker
```
```
/var/lib/docker# sudo du -sm aufs/*
13782 aufs/diff
5 aufs/layers
5 aufs/mnt
```
``` shell
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 1.12.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: xfs
Dirs: 1122
```
Same issue here. I understand 1.13 may get data management commands but in the meantime, I just want to safely delete the contents of this directory without killing Docker.
This is relatively blocking at this point.
Same here. Still no official solution?
I've brought this up a few different times in (Docker Community) Slack. Each time a handful of people run through a list of garbage collection scripts/cmds I should run as a solution.
While those have helped (read: not solved - space is still creeping towards full) in the interim, I think we can all agree that's not the ideal long term fix.
@jadametz 1.13 has `docker system prune`.
Beyond that, I'm not sure how else Docker can help (open to suggestion). The images aren't just getting to the system on their own, but rather through pulls, builds, etc.
In terms of actual orphaned layers (no images on the system referencing them), we'll need to address that separately.
I have exactly the same issue!
```docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 2501
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-96-generic
Operating System: Ubuntu 14.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 14.69 GiB
Name: ip-172-31-45-4
ID: R5WV:BXU5:AV6T:GZUK:SAEA:6E74:PRSO:NQOH:EPMQ:W6UT:5DU4:LE64
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
```
No images, containers or volumes. 42Gb in aufs/diff
Anything to help clear this directory safely would be very useful! Tried everything in this thread without any success. Thanks.
@adamdry only third-party script: https://github.com/docker/docker/issues/22207#issuecomment-252560212
Thanks @kayrus I did indeed try that and it increased my total disk usage slightly and didn't appear to do anything to the aufs/diff directory.
I also tried `docker system prune` which didn't run. And I tried `docker rmi $(docker images -qa -f dangling=true)` which didn't find any images to remove.
For anyone interested I'm now using this to clean down all containers, images, volumes and old aufs:
`### FYI I am a Docker noob so I don't know if this causesany underlying issues but it does work for me - use at your own risk ###`
Lots of inspiration taken from here: http://stackoverflow.com/questions/30984569/error-error-creating-aufs-mount-to-when-building-dockerfile
```
docker rm -f $(docker ps -a -q) || docker rmi -f $(docker images -q) || docker rmi -f $(docker images -a -q)
service docker stop
rm -rf /var/lib/docker/aufs
rm -rf /var/lib/docker/image/aufs
rm -f /var/lib/docker/linkgraph.db
service docker start
```
@adamdry Best to not use `-f` when doing rm/rmi as it will hide errors in removal.
I do consider the current situation... where `-f`hides an error and then we are left with some left-over state that is completely invisible to the user... as a bug.
I'm also seeing this on a completely new and unsurprising installation:
```
root@builder:/var/lib/docker# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.4
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 63
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: overlay host null bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.625 GiB
Name: builder
ID: 2WXZ:BT74:G2FH:W7XD:VVXM:74YS:EA3A:ZQUK:LPID:WYKF:HDWC:UKMJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Insecure Registries:
127.0.0.0/8
root@builder:/var/lib/docker# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
root@builder:/var/lib/docker# docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
root@builder:/var/lib/docker# du -hd2
4.0K ./swarm
6.0M ./image/aufs
6.0M ./image
4.0K ./trust
28K ./volumes
4.0K ./containers
276K ./aufs/layers
292K ./aufs/mnt
1.5G ./aufs/diff |-------------------------
1.5G ./aufs
4.0K ./tmp
72K ./network/files
76K ./network
1.5G .
root@builder:/var/lib/docker#
```
@robhaswell Seeing as it's a new install, do you want to try this? https://github.com/docker/docker/issues/22207#issuecomment-266784433
@adamdry I've already deleted `/var/lib/docker/aufs` as it was blocking my work. What do you expect your instructions to achieve? If they stop the problem from happening again in the future, then I can try to recreate the issue and try your instructions. However if the purpose is just to free up the space then I've already achieved that.
@robhaswell Yeah it was to free up disk space but I had follow up issues when trying to rebuild my images but by following all the steps in that script it resolved those issues.
During build, if the build process is interrupted during layer build process (which also contains a blob to be copied), followed by stopping the container, it leaves behind data in /var/lib/docker/aufs/diff/. It showed up a dangling image. Cleaning itup too didn't release the space. Is it possible to include it as a part of docker system prune? Only deleting the blob data inside this folder frees up the space, which I am not sure will cause any issue or not.
Docker version : 1.13.0-rc1
| During build, if the build process is interrupted during layer build process (which also contains a blob to be copied), followed by stopping the container, it leaves behind data
This could also be the cause of my problems - I interrupt a lot of builds.
During docker pull, observed the following two cases:
1. if the process is interrupted when it says download (which downloads the image layer in /var/lib/docker/tmp/) , it cleans up all the data in that folder
2. If the process is interrupted when it says extracting(which i suppose is extracting the layer from tmp to /var/lib/docker/aufs/diff/), it cleans up the tmp and diff blob data both.
During image build process,
1. On interrupting the process when "Sending build context to docker daemon" ( which copies blob data in my case in /var/lib/docker/tmp/), it remains there forever and cannot be cleaned by any command except manually deleting it. I am not sure how the apt get updates in image are handled.
2. While the layer is being built which contains a blob data, say a large software setup, if the process is interrupted, the docker container keeps working on the image. In my case only 1 layer blob data, which is already available in tmp folder, makes up the whole image. But, if the container is stopped using the docker stop command, two cases happen:
a. if the mount process is still happening, it will leave behind data in tmp and diff folder.
b. If the data copied in diff folder, it will remove the data from tmp folder and leave data in diff folder and maybe mount folder.
We have an automated build process, which needs a control to stop any build process gracefully. Recently, a process got killed by kernel due to out of memory error on a machine which was of low configuration.
If one image is to be built of 2 layers, 1 layer is built and 2nd is interrupted, Docker system prune seems to clean up the data for the container of layer which was interrupted and container stopped. But it doesn't clean up the data of the previous layers in case of interrupt. Also, it didn't reflect the total disk space claimed. Ran these tests on AWS, ubuntu 14.04, x86_64 bit systemwith aufs filesystem. Ran the docker prune test with docker 1.13.0 rc3 and docker 1.12
@thaJeztah
Please let me know if i am misinterpreting anything.
I opened an issue for the `/var/lib/docker/tmp` files not being cleaned up; https://github.com/docker/docker/issues/29486
| Docker system prune seems to clean up the data for the container of layer which was interrupted and container stopped. But it doesn't clean up the data of the previous layers in case of interrupt.
I tried to reproduce that situation, but wasn't able to see that with a simple case;
Start with a clean install empty `/var/lib/docker`, create a big file for
testing, and a Dockerfile;
```bash
mkdir repro || cd repro
fallocate -l 300M bigfile
```
```Dockerfile
cat | Dockerfile ||EOF
FROM scratch
COPY ./bigfile /
COPY ./bigfile /again/
COPY ./bigfile /and-again/
EOF
```
start `docker build`, and cancel while building, but _after_ the build
context has been sent;
```bash
docker build -t stopme .
Sending build context to Docker daemon 314.6 MB
Step 1/4 : FROM scratch
---|
Step 2/4 : COPY ./bigfile /
---| 28eb6d7b0920
Removing intermediate container 98876b1673bf
Step 3/4 : COPY ./bigfile /again/
^C
```
check content of `/var/lib/docker/aufs/`
```bash
du -h /var/lib/docker/aufs/
301M /var/lib/docker/aufs/diff/9127644c356579741348f7f11f50c50c9a40e0120682782dab55614189e82917
301M /var/lib/docker/aufs/diff/81fd6b2c0cf9a28026cf8982331016a6cd62b7df5a3cf99182e7e09fe0d2f084/again
301M /var/lib/docker/aufs/diff/81fd6b2c0cf9a28026cf8982331016a6cd62b7df5a3cf99182e7e09fe0d2f084
601M /var/lib/docker/aufs/diff
8.0K /var/lib/docker/aufs/layers
4.0K /var/lib/docker/aufs/mnt/9127644c356579741348f7f11f50c50c9a40e0120682782dab55614189e82917
4.0K /var/lib/docker/aufs/mnt/81fd6b2c0cf9a28026cf8982331016a6cd62b7df5a3cf99182e7e09fe0d2f084
4.0K /var/lib/docker/aufs/mnt/b6ffb1d5ece015ed4d3cf847cdc50121c70dc1311e42a8f76ae8e35fa5250ad3-init
16K /var/lib/docker/aufs/mnt
601M /var/lib/docker/aufs/
```
run the `dockersystem prune` command to clean up images, containers;
```
docker system prune -a
WARNING! This will remove:
- all stopped containers
- all volumes not used by at least one container
- all networks not used by at least one container
- all images without at least one container associated to them
Are you sure you want to continue? [y/N] y
Deleted Images:
deleted: sha256:253b2968c0b9daaa81a58f2a04e4bc37f1dbf958e565a42094b92e3a02c7b115
deleted: sha256:cad1de5fd349865ae10bfaa820bea3a9a9f000482571a987c8b2b69d7aa1c997
deleted: sha256:28eb6d7b09201d58c8a0e2b861712701cf522f4844cf80e61b4aa4478118c5ab
deleted: sha256:3cda5a28d6953622d6a363bfaa3b6dbda57b789e745c90e039d9fc8a729740db
Total reclaimed space: 629.1 MB
```
check content of `/var/lib/docker/aufs/`
```bash
du -h /var/lib/docker/aufs/
4.0K /var/lib/docker/aufs/diff
4.0K /var/lib/docker/aufs/layers
4.0K /var/lib/docker/aufs/mnt/b6ffb1d5ece015ed4d3cf847cdc50121c70dc1311e42a8f76ae8e35fa5250ad3-init
8.0K /var/lib/docker/aufs/mnt
20K /var/lib/docker/aufs/
```
I do see that the `-init` mount is left behind, I'll check if we can solve
that (although it's just an empty directory)
The only difference in the dockerfile I had used was ( to create different layers)
FROM scratch
COPY ["./bigfile", "randomNoFile1", /]
COPY ["./bigfile", "randomNoFile2", /]
EOF
I am not sure if it makes a difference.
No, the problem isn't about the empty init folders. In my case, it was te blob. However, i can recheck it on monday and update.
Also, was using a 5GB file, created it by reading bytes from dev urandom.
In your case, the same file is added 2 times. Would that create single layer and mount the 2nd layer from it or would it be 2 separate layers? In my case, its always 2 separate layers.
@thaJeztah
Thank you for such a quick response on the issue. Addition of this feature would be of great help!
@monikakatiyar16 I tried to reproduce this as well with canceling the build multiple times during both `ADD` and `RUN` commands but couldn't get anything to leak to `aufs/diff` after deletion. I couldn't quite understand what container you are stopping because containers should not be running during `ADD/COPY` operations. If you can put together a reproducer that we could run that would be greatly appreciated.
Its possible that I could be doing something wrong. Since I am travelling on weekend, I will reproduce it and update all the required info here on Monday.
@tonistiigi @thaJeztah
I feel you are right. There are actually no containers that are listed as active and running. Instead there are dead containers. Docker system prune didn't work in my case, might be since the process didn't get killed with Ctrl+C. Instead, it kept running at the background. In my case, that would be the reason, it couldn't remove those blobs.
When I interrupt the process using Ctrl+C, the build process gets killed, but a process for docker-untar remains alive in the background, which keeps working on building the image. (Note: /var/lib/docker is soft linked to /home/lib/docker to use the EBS volumes for large data on AWS)
`root 12700 10781 7 11:43 ? 00:00:04 docker-untar /home/lib/docker/aufs/mnt/d446d4f8a7dbae162e7578af0d33ac38a63b4892905aa86a8d131c1e75e2828c`
I have attached the script I had been using for creating large files and building the image (gc_maxpush_pull.sh)
Also attached the behaviour of build process for building an image-interrupting it with Ctrl+C (DockerBuild_WOProcessKill) and building image -interrupting it with Ctrl+C - killing the process (DockerBuild_WithProcessKill)
Using the commands -
To create large file : `./gc_maxpush_pull.sh 1 5gblayer 0 512 1`
To build images : `./gc_maxpush_pull.sh 1 5gblayer 1 512 1`
[DockerBuild.zip](https://github.com/docker/docker/files/660942/DockerBuild.zip)
Steps to replicate :
1. Create a large file of 5GB
2. Start the build process and interrupt it only after Sending build context is over and it's actually copying the blob.
3. It completes building the image after a while and shows it up in docker images (as in case 1 attached by me - DockerBuild_WOProcessKill)
4. If the process is killed, it takes a while and leaves the blob data in /diff (which it should on killing process abruptly as attached in file - DockerBuild_WithProcessKill)
If what I am assuming is correct, then this might not be an issue with docker prune, instead with killing of docker build that is somehow not working for me.
Is there a graceful way of interrupting or stopping the build image process, which also takes care of cleaning up the copied data (as handled in docker pull)?
Previously, I was not killing the process. I am also curious what docker-untar does and why it mounts it to /mnt and /diff folders both and later clean out /mnt folder?
Tested this with Docker version 1.12.5, build 7392c3b on AWS
docker info
Containers: 2
Running: 0
Paused:0
Stopped: 2
Images: 0
Server Version: 1.12.5
Storage Driver: aufs
Root Dir: /home/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 4
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: overlay bridge null host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-105-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859 GiB
Name: master
ID: 2NQU:D2C5:5WPL:IIDR:P6FO:OAG7:GHW6:ZJMQ:VDHI:B5CI:XFZJ:ZSZM
Docker Root Dir: /home/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
@monikakatiyar16 When I manually kill `untar` process during build I get `Error processing tar file(signal: killed):` in the build output. Leaving behind the container in `docker ps -a` is the correct behavior, same thing happens in any build error and lets you debug the problems that caused the build to fail. I have no problem with deleting this container though and if I do that all data in `/var/lib/docker/aufs` is cleaned up as well.
@tonistiigi Yes you are correct. I was able to delete the volume associated with the container and it cleaned up everything, after killing the docker-untar process. Docker system prune also works in this case.
The actual issue which left over volumes was the case, when without killing the docker-untar process, i triedremoving the docker container along with the volumes - which gave the following error :
`docker rm -v -f $(docker ps -a -q)`
`Error response from daemon: Driver aufs failed to remove root filesystem 97931bf059a0ec219efd3f762dbb173cf9372761ff95746358c08e2b61f7ce79: rename /home/lib/docker/aufs/diff/359d27c5b608c9dda1170d1e34e5d6c5d90aa2e94826257f210b1442317fad70 /home/lib/docker/aufs/diff/359d27c5b608c9dda1170d1e34e5d6c5d90aa2e94826257f210b1442317fad70-removing: device or resource busy`
Daemon logs:
`Error removing mounted layer 78fb899aab981557bc2ee48e9738ff4c2fcf2d10a1984a62a77eefe980c68d4a: rename /home/lib/docker/aufs/diff/d2605125ef072de79dc948f678aa94dd6dde562f51a4c0bd08a210d5b2eba5ec /home/lib/docker/aufs/diff/d2605125ef072de79dc948f678aa94dd6dde562f51a4c0bd08a210d5b2eba5ec-removing: device or resource busy
ERRO[0956] Handler for DELETE /v1.25/containers/78fb899aab98 returned error: Driver aufs failed to remove root filesystem 78fb899aab981557bc2ee48e9738ff4c2fcf2d10a1984a62a77eefe980c68d4a: rename /home/lib/docker/aufs/diff/d2605125ef072de79dc948f678aa94dd6dde562f51a4c0bd08a210d5b2eba5ec /home/lib/docker/aufs/diff/d2605125ef072de79dc948f678aa94dd6dde562f51a4c0bd08a210d5b2eba5ec-removing: device or resource busy
ERRO[1028] Error unmounting container 78fb899aab981557bc2ee48e9738ff4c2fcf2d10a1984a62a77eefe980c68d4a: no such file or directory`
It seems that the order right now to be followed to interrupt a docker build is :
`Interrupt docker build | Kill docker untar process | remove container and volume : docker rm -v -f $(docker ps -a -q)`
For `docker v1.13.0-rc4`, it can be `Interrupt docker build | Kill docker untar process | docker system prune -a`
This seems to work perfectly. There are no issues of cleanup, instead the only issue is the docker-untar process not being killed along with the docker-build process.
I will search/update/log a new issue for graceful interrupt of docker build that also stops docker-untar process along with it.
(Verified this with docker v1.12.5 and v1.13.0-rc4)
Update : On killing docker-untar while Sending build context to docker daemon, it gives an error in build : `Error response from daemon: Error processing tar file(signal: terminated)` , but during layer copy it doesn't (for me)
Thanks for being so patient andfor giving your time!
I'm seeing `/var/lib/docker/aufs` consistently increase in size on a docker swarm mode worker. This thing is mostly autonomous being managed by the swarm manager and very little manual container creation aside from some maintenance commands here and there.
I do run `docker exec` on service containers; not sure if that may be a cause.
My workaround to get this resolved in my case was tostart up another worker, set the full node to `--availability=drain` and manually move over a couple of volume mounts.
```
ubuntu@ip-172-31-18-156:~$ docker --version
Docker version 1.12.3, build 6b644ec
```
This has hit our CI server for ages. This needs to be fixed.
@orf thanks
Same issue here. Neither container, volumes and image removing, nore Docker 1.13 cleanup commands have any effect.
I also confirm I did some image build cancels. Maybe that leaves folders that can't be reacher either.
I'll use the good old rm method for now, but this is clearly a bug.
Files in the /var/lib/docker/aufs/diff fills up 100% space for /dev/sda1 filesystem of 30G
root@Ubuntu:/var/lib/docker/aufs/diff# df -h
Filesystem Size Used Avail Use% Mounted on
udev 14G 0 14G 0% /dev
tmpfs 2.8G 273M 2.5G 10% /run
**/dev/sda1 29G 29G 0 100% /**
tmpfs 14G 0 14G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 14G 0 14G 0% /sys/fs/cgroup
/dev/sdb1 197G 60M 187G 1% /mnt
tmpfs 2.8G 0 2.8G 0% /run/user/1000
du -h -d 1 /var/lib/docker/aufs/diff | grep '[0-9]G\|'
shows
4.1G /var/lib/docker/aufs/diff/a0cde42cbea362bbb2a73ffbf30059bcce7ef0256d1d7e186264f915d15
14G /var/lib/docker/aufs/diff/59aee33d8a607b5315ce103cd99f17b4dfdec73c9a2f3bb2afc7d02bfae
20G /var/lib/docker/aufs/diff
Also tried, **docker system prune**, that did not help.
Has anyone found a solution for this ongoing issue of super large files in diff before this bug is fixed in the code?
Yes, the method has alreadybeen given, but here is a apocalypse snippet that just destroy everything I put in place here at work (except local folders for the volumes). To put in the bashrc or another bash config file.
```
alias docker-full-cleanup='func_full-cleanup-docker'
func_full-cleanup-docker() {
echo "WARN: This will remove everything from docker: volumes, containers and images. Will you dare? [y/N] "
read choice
if [ \( "$choice" == "y" \) -o \( "$choice" == "Y" \) ]
then
sudo echo "| sudo rights check [OK]"
sizea=`sudo du -sh /var/lib/docker/aufs`
echo "Stopping all running containers"
containers=`docker ps -a -q`
if [ -n "$containers" ]
then
docker stop $containers
fi
echo "Removing all docker images and containers"
docker system prune -f
echo "Stopping Docker daemon"
sudo service docker stop
echo "Removing all leftovers in /var/lib/docker (bug #22207)"
sudo rm -rf /var/lib/docker/aufs
sudo rm -rf /var/lib/docker/image/aufs
sudo rm -f /var/lib/docker/linkgraph.db
echo "Starting Docker daemon"
sudo service docker start
sizeb=`sudo du -sh /var/lib/docker/aufs`
echo "Size before full cleanup:"
echo " $sizea"
echo "Size after full cleanup:"
echo " $sizeb"
fi
}```
I ran rm -rf command to remove the files from the diff folder for now. Probably will have to look into the script if the diff folder occupies the entire dis space again.
Hope to see this issue fixed in the code, instead of work arounds.
Hi, I have same issue in docker 1.10.2, I'm running kubernetes. this is my docker version:
```
Containers: 7
Running: 0
Paused: 0
Stopped: 7
Images: 4
Server Version: 1.10.2
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 50
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 4.4.0-31-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.954 GiB
Name: ubuntu-k8s-03
ID: NT23:5Y7J:N2UM:NA2W:2FHE:FNAS:56HF:WFFF:N2FR:O4T4:WAHC:I3PO
Debug mode (server): true
File Descriptors: 10
Goroutines: 23
System Time: 2017-02-14T15:25:00.740998058+09:00
EventsListeners: 0
Init SHA1: 3e247d0d32543488f6e70fbb7c806203f3841d1b
Init Path: /usr/lib/docker/dockerinit
Docker Root Dir: /var/lib/docker
WARNING: No swap limit support
```
I'm trying to track all unused diff directory under `/var/lib/docker/aufs/diff` and `/var/lib/docker/aufs/mnt/` by analyzing layer files under `/var/lib/docker/image/aufs/imagedb`, here is the script I used:
https://gist.github.com/justlaputa/a50908d4c935f39c39811aa5fa9fba33
But I met problem when I stop and restart the docker daemon, seems I make some inconsistent status of docker:
/var/log/upstart/docker.log:
```
DEBU[0277] Cleaning up old shm/mqueue mounts: start.
DEBU[0277] Cleaning up old shm/mqueue mounts: done.
DEBU[0277] Clean shutdown succeeded
Waiting for /var/run/docker.sock
DEBU[0000] docker group found. gid: 999
DEBU[0000] Server created for HTTP on unix (/var/run/docker.sock)
DEBU[0000] Using default logging driver json-file
INFO[0000] [graphdriver] using prior storage driver "aufs"
DEBU[0000] Using graph driver aufs
INFO[0000] Graph migration to content-addressability took 0.00 seconds
DEBU[0000] Option DefaultDriver: bridge
DEBU[0000] Option DefaultNetwork: bridge
INFO[0000] Firewalld running: false
DEBU[0000] /sbin/iptables, [--wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -D PREROUTING]
DEBU[0000] /sbin/iptables, [--wait -t nat -D OUTPUT]
DEBU[0000] /sbin/iptables, [--wait -t nat -F DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -X DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -F DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -X DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION]
DEBU[0000] /sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION]
DEBU[0000] /sbin/iptables, [--wait -t nat -n -L DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -N DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -n -L DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t filter -n -L DOCKER-ISOLATION]
DEBU[0000] /sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION -j RETURN]
DEBU[0000] /sbin/iptables, [--wait -I DOCKER-ISOLATION -j RETURN]
/var/run/docker.sock is up
DEBU[0000] Registering ipam driver: "default"
DEBU[0000] releasing IPv4 pools from network bridge (dcfcc71060f02440ae53da5ee0f083ca51c33a290565f1741f451754ae6b4257)
DEBU[0000] ReleaseAddress(LocalDefault/10.254.69.0/24, 10.254.69.1)
DEBU[0000] ReleasePool(LocalDefault/10.254.69.0/24)
DEBU[0000] Allocating IPv4 pools for network bridge (159d0a404ff6564b4fcfe633f0c8c123c0c0606d28ec3b110272650c5fc1bcb6)
DEBU[0000] RequestPool(LocalDefault, 10.254.69.1/24, , map[], false)
DEBU[0000] RequestAddress(LocalDefault/10.254.69.0/24, 10.254.69.1, map[RequestAddressType:com.docker.network.gateway])
DEBU[0000] /sbin/iptables, [--wait -t nat -C POSTROUTING -s 10.254.69.0/24 ! -o docker0 -j MASQUERADE]
DEBU[0000] /sbin/iptables, [--wait -t nat -C DOCKER -i docker0 -j RETURN]
DEBU[0000] /sbin/iptables, [--wait -t nat -I DOCKER -i docker0 -j RETURN]
DEBU[0000] /sbin/iptables, [--wait -D FORWARD -i docker0 -o docker0 -j DROP]
DEBU[0000] /sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT]
DEBU[0000] /sbin/iptables, [--wait -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT]
DEBU[0000] /sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT]
DEBU[0001] /sbin/iptables, [--wait -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0001] /sbin/iptables, [--wait -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0001] /sbin/iptables, [--wait -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8]
DEBU[0001] /sbin/iptables, [--wait -t nat -A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8]
DEBU[0001] /sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -j DOCKER]
DEBU[0001] /sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -j DOCKER]
DEBU[0001] /sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-ISOLATION]
DEBU[0001] /sbin/iptables, [--wait -D FORWARD -j DOCKER-ISOLATION]
DEBU[0001] /sbin/iptables, [--wait -I FORWARD -j DOCKER-ISOLATION]
WARN[0001] Your kernel does not support swap memory limit.
DEBU[0001] Cleaning up old shm/mqueue mounts: start.
DEBU[0001] Cleaning up old shm/mqueue mounts: done.
DEBU[0001] Loaded container 0790b33ec8e5345ac944d560263b8e13cb75f80dd82cd25753c7320bbcb2747c
DEBU[0001] Loaded container 0e36a6c9319e6b7ca4e5b5408e99d77d51b1f4e825248c039ba0260e628c483d
DEBU[0001] Loaded container 135fb2e8cad26d531435dcd19d454e41cf7aece289ddc7374b4c2a984f8b094a
DEBU[0001] Loaded container 2c28de46788ce96026ac8e61e99c145ec55517543e078a781e8ce6c8cddec973
DEBU[0001] Loaded container 35eb075b5815e621378eb8a7ff5ad8652819ec851eaa4f7baedb1383dfa51a57
DEBU[0001] Loaded container 6be37a301a8f52040adf811041c140408224b12599aa55155f8243066d2b0b69
DEBU[0001] Loaded container d98ac7f052fef31761b82ab6c717760428ad5734df4de038d80124ad5b5e8614
DEBU[0001] Starting container 2c28de46788ce96026ac8e61e99c145ec55517543e078a781e8ce6c8cddec973
ERRO[0001] Couldn't run auplink before unmount: exit status 22
ERRO[0001] error locating sandbox id d4c538661db2edc23c79d7dddcf5c7a8886c9477737888a5fc2641bc5e66da8b: sandbox d4c538661db2edc23c79d7dddcf5c7a8886c9477737888a5fc2641bc5e66da8b not found
WARN[0001] failed to cleanup ipc mounts:
failed to umount /var/lib/docker/containers/2c28de46788ce96026ac8e61e99c145ec55517543e078a781e8ce6c8cddec973/shm: invalid argument
ERRO[0001] Failed to start container 2c28de46788ce96026ac8e61e99c145ec55517543e078a781e8ce6c8cddec973: error creating aufs mount to /var/lib/docker/aufs/mnt/187b8026621da2add42330c9393a474fcd9af2e4567596d61bcd7a40c85f71da: invalid argument
INFO[0001] Daemon has completed initialization
INFO[0001] Docker daemon commit=c3959b1 execdriver=native-0.2 graphdriver=aufs version=1.10.2
DEBU[0001] Registering routers
DEBU[0001] Registering HEAD, /containers/{name:.*}/archive
```
and when I try to create new containers by `docker run`, it failed with message:
```
docker: Error response from daemon: error creating aufsmount to /var/lib/docker/aufs/mnt/f9609c0229baa2cdc6bc07c36970ef4f192431c1b1976766b3ea23d72c355df3-init: invalid argument.
See 'docker run --help'.
```
and the daemon log shows:
```
DEBU[0173] Calling POST /v1.22/containers/create
DEBU[0173] POST /v1.22/containers/create
DEBU[0173] form data: {"AttachStderr":false,"AttachStdin":false,"AttachStdout":false,"Cmd":["/hyperkube","kubelet","--api-servers=http://localhost:8080","--v=2","--address=0.0.0.0","--enable-server","--hostname-override=172.16.210.87","--config=/etc/kubernetes/manifests-multi","--cluster-dns=10.253.0.10","--cluster-domain=cluster.local","--allow_privileged=true"],"Domainname":"","Entrypoint":null,"Env":[],"HostConfig":{"Binds":["/sys:/sys:ro","/dev:/dev","/var/lib/docker/:/var/lib/docker:rw","/var/lib/kubelet/:/var/lib/kubelet:rw","/var/run:/var/run:rw","/etc/kubernetes/manifests-multi:/etc/kubernetes/manifests-multi:ro","/:/rootfs:ro"],"BlkioDeviceReadBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceWriteIOps":null,"BlkioWeight":0,"BlkioWeightDevice":null,"CapAdd":null,"CapDrop":null,"CgroupParent":"","ConsoleSize":[0,0],"ContainerIDFile":"","CpuPeriod":0,"CpuQuota":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","Devices":[],"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IpcMode":"","Isolation":"","KernelMemory":0,"Links":null,"LogConfig":{"Config":{},"Type":""},"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NetworkMode":"host","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"host","PidsLimit":0,"PortBindings":{},"Privileged":true,"PublishAllPorts":false,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"always"},"SecurityOpt":null,"ShmSize":0,"UTSMode":"","Ulimits":null,"VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"gcr.io/google_containers/hyperkube:v1.1.8","Labels":{},"NetworkingConfig":{"EndpointsConfig":{}},"OnBuild":null,"OpenStdin":false,"StdinOnce":false,"StopSignal":"SIGTERM","Tty":false,"User":"","Volumes":{},"WorkingDir":""}
ERRO[0173] Couldn't run auplink before unmount: exit status 22
ERRO[0173] Clean up Error! Cannot destroy container 482957f3e4e92a0ba56d4787449daa5a8708f3b77efe0c603605f35d02057566: nosuchcontainer: No such container: 482957f3e4e92a0ba56d4787449daa5a8708f3b77efe0c603605f35d02057566
ERRO[0173] Handler for POST /v1.22/containers/create returned error: error creating aufs mount to /var/lib/docker/aufs/mnt/f9609c0229baa2cdc6bc07c36970ef4f192431c1b1976766b3ea23d72c355df3-init: invalid argument
```
does anyone know whether my approach is correct or not? and why the problem happens after I delete those folders?
I opened #31012 to at least make sure we don't leak these dirs in any circumstances.
We of course also need to look at the various causes of the `busy` errors
This was biting me as long as I can remember. I got pretty much the same results as described above when I switched to `overlay2` driver some days ago and nuked the aufs folder completely (`docker system df` says 1.5Gigs, `df` says 15Gigs).
I had about 1T of diffs using storage. After restarting my docker daemon - I recovered about 700GB. So I guess stopping the daemon prunes these?
Restarting does nothing for me, unfortunately.
Service restart did not help. This is a serious issue. Removing all container/images does not remove those diffs.
Stopping the daemon would not prune these.
If you remove all containers and you still have `diff` dirs, then likely you have some leaked rw layers.
We just encountered this issue. `/var/lib/docker/aufs/diff` took up 28G and took our root filesystem to 100%, which caused our GitLab server to stop responding. We're using docker for GitLab CI. To fix this, I used some of the commands @sogetimaitral suggested above to delete the temp files, and we're back up and running. I restarted the server and sent in a new commit to trigger CI, and everything appears to be working just as it should.
I'm definitely concerned this is going to happen again. What's the deal here? Is this a docker bug that needs to be fixed?
1. Yes there is a bug (both that there are issues on removal and that --force on rm ignores these issues)
2. Generally one should not be writing lots of data to the container fs and instead use a volume (even a throw-away volume). A large diff dir would indicate that there is significant amounts of data being written to the container fs.
If you don't use "--force" on remove you would not run into this issue (or at least you'd see you have a bunch of "dead" containers and know how/what to clean up.).
I'm not manually using docker at all. We're using [gitlab-ci-multi-runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner). Could it be a bug on GitLab's end then?
It looks like (by default) it force-removes containers; https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/dbdbce2848530df299836768c8ea01e209a2fe40/executors/docker/executor_docker.go#L878. Doing so can result in failures to remove the container being ignored, and leading to the orphaned diffs.
Ok, then that tells me that this is a gitlab-ci-multi-runner bug. Is that a correct interpretation? I'm happy to create an issue for them to fix this.
It's a combination I guess; "force" remove makes it easier to handle cleanups (i.e., cases where a container isn't stopped yet, etc), at the same time (that's the "bug" @cpuguy83 mentioned), it can also hide actual issues, such as docker failing to remove the containers filesystem (which can have various reasons). With "force", the container is removed in such cases. Without, the container is left around (but marked "dead")
If the gitlab runner can function correctly without the force remove, that'll probably be good to change (or make it configurable)
I am using [Drone](https://github.com/drone/drone) and have the same issue. I didn't check the code how containers are removed, but i guess it force removes as well.
Could it be a Docker in Docker issue? I am starting Drone with docker-compose.
I decided to go ahead and submit a gitlab-ci-multi-runner issue just to loop the devs in: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/2304
Honestly we worked around this by running Spotify's docker gc with drone CI.
El El mar, mar. 28, 2017 a las 3:38 PM, Geoffrey Fairchild |
notifications@github.com| escribió:
| I decided to go ahead and submit a gitlab-ci-multi-runner issue just to
| loop the devs in:
| https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/2304
|
| —
| You are receiving this because you commented.
| Reply to this email directly, view it on GitHub
| |https://github.com/docker/docker/issues/22207#issuecomment-289926298|,
| or mute the thread
| |https://github.com/notifications/unsubscribe-auth/AC6Wz197zkjWWOlq1-JOibiQP-xJym9Eks5rqYvegaJpZM4IMGt2|
| .
|
@sedouard Thanks for this tip! Running [docker-gc](https://github.com/spotify/docker-gc) hourly from spotify solved the problem for me.
We are getting this issue running from Gitlab CI (not running in docker), using commands to build images / run containers, (not Gitlab CI Docker integration). We are not running any form of force removal, simply `docker run --rm ...` and `docker rmi image:tag`
**EDIT**: sorry, actually the original problem is the same. The difference is that running `spotify/docker-gc` does _not_ fix the problem.
----
As you can see below, I have 0 images, 0 containers, nothing!
`docker system info` agrees with me, but mentions `Dirs: 38` for the aufs storage.
That's suspicious! If you look at `/var/lib/docker/aufs/diff/`, we see that there's actually 1.7 GB of data there, over 41 directories. And that's my personal box, on the production server it's 19 GB.
How do we clean this? using `spotify/docker-gc` does not remove these.
``` shell
philippe@pv-desktop:~$ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
philippe@pv-desktop:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
philippe@pv-desktop:~$ docker system info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 17.03.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 38
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-72-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 31.34 GiB
Name: pv-desktop
ID: 2U5D:CRHS:RUQK:YSJX:ZTRS:HYMV:HO6Q:FDKE:R6PK:HMUN:2EOI:RUWO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: silex
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
philippe@pv-desktop:~$ ls -alh /var/lib/docker/aufs/diff/
total 276K
drwxr-xr-x 40 root root 116K Apr 13 15:32 .
drwxr-xr-x 5 root root 4.0K Sep 18 2015 ..
drwxr-xr-x 4 root root 4.0K Jun 17 2016 005d00efb0ba949d627ad439aec8c268b5d55759f6e92e51d7828c12e3817147
drwxr-xr-x 8 root root 4.0K May 2 2016 0968e52874bbfaa938ffc869cef1c5b78e2d4f7a670e19ef47f713868b9bfbdf
drwxr-xr-x 4 root root 4.0K Jun 20 2016 188233e6dcc37e2308e69807ffd19aca3e61be367daae921f2bcb15a1d6237d0
drwxr-xr-x 6 root root 4.0K Jun 20 2016 188233e6dcc37e2308e69807ffd19aca3e61be367daae921f2bcb15a1d6237d0-init
drwxr-xr-x 21 root root 4.0K Apr 8 2016 250ecb97108a6d8a8c41f9d2eb61389a228c95f980575e95ee61f9e8629d5180
drwxr-xr-x 2 root root 4.0K Dec 22 2015 291f16f99d9b0bc05100e463dbc007ef816e0cf17b85d20cf51da5eb2b866810
drwxr-xr-x 2 root root 4.0K May 2 2016 3054baaa0b4a7b52da2d25170e9ce4865967f899bdf6d444b571e57be141b712
drwxr-xr-x 2 root root 4.0K Feb 5 2016 369aca82a5c05d17006b9dca3bf92d1de7d39d7cd908ed665ef181649525464e
drwxr-xr-x 3 root root 4.0K Jun 17 2016 3835a1d1dfe755d9d1ada6933a0ea7a4943caf8f3d96eb3d79c8de7ce25954d2
(...strip)
philippe@pv-desktop:~$ du -hs /var/lib/docker/aufs/diff/
1.7G /var/lib/docker/aufs/diff/
philippe@pv-desktop:~$ docker system prune -a
WARNING! This will remove:
- all stopped containers
- all volumes not used by at least one container
- all networks not used by at least one container
- all images without at least one container associated to them
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0 B
```
Can I safely `rm -r /var/lib/docker/aufs` and restart the docker deamon?
Running `spotify/docker-gc` does not clean those orphans.
**EDIT**: thanks @CVTJNII!
Stopping the Docker daemon and erasing all of /var/lib/docker will be safer. Erasing /var/lib/docker/aufs will cause you to lose your images anyway so it's better to start with a clean /var/lib/docker in my opinion. This is the "solution" I've been using for several months for this problem now.
Starting with 17.06 there should no longer be any *new* orphaned diffs.
Instead you may start seeing containers with the state `Dead`, this happens if there was an error during removal that is non-recoverable and may require an admin to deal with it.
In addition, removal is a bit more robust, and less prone to error due to race conditions or failed unmounts.
@cpuguy83: great news, can you explain what the admin would need to do if that happens?
@Silex It depends on the cause.
Typically what has happened is there is a `device or resource busy` error due to some mount being leaked into a container. If you are running something like cadvisor this is pretty much a guarantee as the instructions say to mount the whole docker dir into the cadvisor container.
This *can* be tricky, you may have to stop the offending container(s) and then remove the `dead` container.
If you are on a newer kernel (3.15+) it is unlikely that you would see the issue anymore, though there still may be some edge case.
Docker version 17.06.0-ce, build 02c1d87
I tried remove all images, volumes, networks and containers but it not helped.
Also tried commands:
```
docker system prune -af
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc:ro spotify/docker-gc
```
Still remain files:
```
root@Dark:/var/lib/docker/aufs# ls -la *
diff:
total 92
drwx------ 12 root root 45056 Jul 28 17:28 .
drwx------ 5 root root 4096 Jul 9 00:18 ..
drwxr-xr-x 4 root root 4096 Jul 10 01:35 78f8ecad2e94fedfb1ced425885fd80bb8721f9fd70715de2ce373986785b882
drwxr-xr-x 6 root root 4096 Jul 10 01:35 78f8ecad2e94fedfb1ced425885fd80bb8721f9fd70715de2ce373986785b882-init
drwxr-xr-x 5 root root 4096 Jul 10 01:35 7caa9688638ea9669bac451b155b65b121e99fcea8d675688f0c76678ba85ccd
drwxr-xr-x 6 root root 4096 Jul 10 01:35 7caa9688638ea9669bac451b155b65b121e99fcea8d675688f0c76678ba85ccd-init
drwxr-xr-x 4 root root 4096 Jul 12 14:45 b7b7770aae461af083e72e5e3232a62a90f934c83e38830d06365108e302e7ac
drwxr-xr-x 6 root root 4096 Jul 12 14:45 b7b7770aae461af083e72e5e3232a62a90f934c83e38830d06365108e302e7ac-init
drwxr-xr-x 4 root root 4096 Jul 10 01:35 d5752b27b341e17e730d3f4acbec04b10e41dc01ce6f9f98ff38208c0647f2e4
drwxr-xr-x 6 root root 4096 Jul 10 01:35 d5752b27b341e17e730d3f4acbec04b10e41dc01ce6f9f98ff38208c0647f2e4-init
drwxr-xr-x 6 root root 4096 Jul 10 01:35 e412d3c6f0f5f85e23d7a396d47c459f5d74378b474b27106ab9b82ea829dbfb
drwxr-xr-x 6 root root 4096 Jul 10 01:35 e412d3c6f0f5f85e23d7a396d47c459f5d74378b474b27106ab9b82ea829dbfb-init
layers:
total 52
drwx------ 2 root root 45056 Jul 28 17:28 .
drwx------ 5 root root 4096 Jul 9 00:18 ..
-rw-r--r-- 1 root root 0 Jul 10 01:35 78f8ecad2e94fedfb1ced425885fd80bb8721f9fd70715de2ce373986785b882
-rw-r--r-- 1 root root 0 Jul 10 01:35 78f8ecad2e94fedfb1ced425885fd80bb8721f9fd70715de2ce373986785b882-init
-rw-r--r-- 1 root root 0 Jul 10 01:35 7caa9688638ea9669bac451b155b65b121e99fcea8d675688f0c76678ba85ccd
-rw-r--r-- 1 root root 0 Jul 10 01:35 7caa9688638ea9669bac451b155b65b121e99fcea8d675688f0c76678ba85ccd-init
-rw-r--r-- 1 root root 0 Jul 12 14:45 b7b7770aae461af083e72e5e3232a62a90f934c83e38830d06365108e302e7ac
-rw-r--r-- 1 root root 0 Jul 12 14:45 b7b7770aae461af083e72e5e3232a62a90f934c83e38830d06365108e302e7ac-init
-rw-r--r-- 1 root root 0 Jul 10 01:35 d5752b27b341e17e730d3f4acbec04b10e41dc01ce6f9f98ff38208c0647f2e4
-rw-r--r-- 1 root root 0 Jul 10 01:35 d5752b27b341e17e730d3f4acbec04b10e41dc01ce6f9f98ff38208c0647f2e4-init
-rw-r--r-- 1 root root 0 Jul 10 01:35 e412d3c6f0f5f85e23d7a396d47c459f5d74378b474b27106ab9b82ea829dbfb
-rw-r--r-- 1 root root 0 Jul 10 01:35 e412d3c6f0f5f85e23d7a396d47c459f5d74378b474b27106ab9b82ea829dbfb-init
mnt:
total 92
drwx------ 12 root root 45056 Jul 28 17:28 .
drwx------ 5 root root 4096 Jul 9 00:18 ..
drwxr-xr-x 2 root root 4096 Jul 10 01:35 78f8ecad2e94fedfb1ced425885fd80bb8721f9fd70715de2ce373986785b882
drwxr-xr-x 2 root root 4096 Jul 10 01:35 78f8ecad2e94fedfb1ced425885fd80bb8721f9fd70715de2ce373986785b882-init
drwxr-xr-x 2 root root 4096 Jul 10 01:35 7caa9688638ea9669bac451b155b65b121e99fcea8d675688f0c76678ba85ccd
drwxr-xr-x 2 root root 4096 Jul 10 01:35 7caa9688638ea9669bac451b155b65b121e99fcea8d675688f0c76678ba85ccd-init
drwxr-xr-x 2 root root 4096 Jul 12 14:45 b7b7770aae461af083e72e5e3232a62a90f934c83e38830d06365108e302e7ac
drwxr-xr-x 2 root root 4096 Jul 12 14:45 b7b7770aae461af083e72e5e3232a62a90f934c83e38830d06365108e302e7ac-init
drwxr-xr-x 2 root root 4096 Jul 10 01:35 d5752b27b341e17e730d3f4acbec04b10e41dc01ce6f9f98ff38208c0647f2e4
drwxr-xr-x 2 root root 4096 Jul 10 01:35 d5752b27b341e17e730d3f4acbec04b10e41dc01ce6f9f98ff38208c0647f2e4-init
drwxr-xr-x 2 root root 4096 Jul 10 01:35 e412d3c6f0f5f85e23d7a396d47c459f5d74378b474b27106ab9b82ea829dbfb
drwxr-xr-x 2 root root 4096 Jul 10 01:35 e412d3c6f0f5f85e23d7a396d47c459f5d74378b474b27106ab9b82ea829dbfb-init
```
```
# docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 0 0 0B 0B
Containers 0 0 0B 0B
Local Volumes 0 0 0B 0B
```
How can it be deleted?
@haos616 try stopping all running containers first, and then run `docker system prune -af`.
This did the trick for me.
Didn't work while I had a container running.
If it's an upgrade from a previous version of docker, it's possible those diffs were generated / left behind by that version. Docker 17.06 won't remove a container if layers failed to be removed (when using --force); older versions did, which could lead to orphaned layers
@julian-pani I did so in the beginning but it does not help.
```
# docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 0 0 0B 0B
Containers 0 0 0B 0B
Local Volumes 0 0 0B 0B
```
@thaJeztah No. I cleaned the Docker one or two months ago. Then the version was already 17.06. I used command `docker system prune -af`. It removed everything.
Running https://github.com/spotify/docker-gc as a container worked for me, but it went a step extra and deleted some of my required images too :(
So I've put a small wrapper script as below to be safe
```
#!/bin/sh
docker images -q | /etc/docker-gc-exclude # Save all genuine images as exclude
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc:ro spotify/docker-gc
```
thanks again to spotify
IIUC, the spotify script just calls `docker rm` and `docker rmi` - did it actually remove orphaned diffs?
Just some feedback for the community, I've read through all of this and none of the solutions actually seem to work consistently or reliably. My "fix" was simply to double the amount of disc space on my AWS instances. And I know all too well that's a crappy fix but it is the best workaround I've found to Docker's bloated aufs. This really, really needs to be fixed.
@fuzzygroup 17.06 should no longer create orphaned diffs, but it won't clean up the old ones yet.
I could cleanup with this script. I don't see why it wouldn't work, but who knows.
Anyway it's working fine for me. It will delete all images, containers, and volumes... As it should not run very often, I find it a minor side effect. But it's up to you to use it or not.
https://gist.github.com/Karreg/84206b9711cbc6d0fbbe77a57f705979
https://stackoverflow.com/q/45798076/562769 seems to be related. I've posted a quick fix.
FYI, still seeing this with `17.06.1-ce`
```
Containers: 20
Running: 0
Paused: 0
Stopped: 20
Images: 124
Server Version: 17.06.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 185
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
apparmor
Kernel Version: 4.4.0-83-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 7.796GiB
Name: gitlab-cirunner
ID: PWLR:R6HF:MK3Y:KN5A:AWRV:KHFY:F36D:WASF:7K7B:U7FY:2DJA:DBE2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
```
`/var/lib/docker/aufs/diff` contains lots of directories with the `-init-removing` and `-removing` prefix:
```
ffd5477de24b0d9993724e40175185038a62250861516030a33280898243e742-init-removing
ffd5477de24b0d9993724e40175185038a62250861516030a33280898243e742-removing
ffd900de0634992e99c022a16775805dfd0ffd1f6c89fece7deb6b1a71c5e38c-init-removing
ffd900de0634992e99c022a16775805dfd0ffd1f6c89fece7deb6b1a71c5e38c-removing
```
| FYI, still seeing this with 17.06.1-ce
Still seeing what, exactly?
There should not be any way that a diff dir can leak, though diff dirs will still exist if they existed on upgrade, they'll still exist.
Still seeing orphaned diffs as far as I can tell. `docker system prune` didn't remove them, neither did `docker-gc`. Manually running `rm -rf /var/lib/docker/aufs/diff/*-removing` seems to be working.
Yes, docker will not clean up old orphaned dirs yet.
By old you mean those created from a previous version of docker with this issue?
This is a fresh install of Docker we did about two weeks ago, those orphans must have been created since then, so it seems that docker must still be creating those orphans?
I mean, in the last half an hour I've got `112` new diffs with `-removing`, since I rm'ed them manually.
```
$ ls /var/lib/docker/aufs/diff/ | grep removing | wc -l
112
```
You said "17.06 should no longer create orphaned diffs, but it won't clean up the old ones yet.", but surely this cannot be correct, or am I missing something? Are those tagged with `-removing` not orphaned?
@orf On a newer kernel, it's highly unexpected to have any issue at all during removal. Are you mounting `/var/lib/docker` into a container?
I'll check in the aufs driver to see if there's a specific issue there with it reporting a successful remove when it really wasn't.
We are not mounting `/var/lib/docker` into a container.
```
$ uname -a
Linux gitlab-cirunner 4.4.0-83-generic #106~14.04.1-Ubuntu SMP Mon Jun 26 18:10:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
```
We are running 14.04 LTS
Let me know if there is anything I can provide to help debug this.
For other reasons (swarm mode networking) I moved off 14.04 for Docker
machines.
On Mon, Aug 21, 2017 at 8:23 AM orf |notifications@github.com| wrote:
| We are not mounting /var/lib/docker into a container.
|
| $ uname -a
| Linux gitlab-cirunner 4.4.0-83-generic #106~14.04.1-Ubuntu SMP Mon Jun 26 18:10:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
|
| We are running 14.04 LTS
|
| —
| You are receiving this because you commented.
| Reply to this email directly, view it on GitHub
| |https://github.com/moby/moby/issues/22207#issuecomment-323773033|, or mute
| the thread
| |https://github.com/notifications/unsubscribe-auth/AADRIfE2B2HNpbsKyTOj1CwGzulRT2C0ks5saaDPgaJpZM4IMGt2|
| .
|
This appears to be worse with 17.06.01-ce. I updated a build machine to this version and immediately started seeing the `*-init-removing` and the `*-removing` directories left around as part of the build process. I stopped the service, removed the `/var/lib/docker` directory, restarted the service and after a few builds was close to out of disk space again. I stopped the service again, ran `apt-get purge docker-ce`, removed `/var/lib/docker` again and installed the 17.06.0-ce version. Not getting the extra directories in `/var/lib/docker/aufs/diff` and disk space is representative of images that are on the build machine. I've reproduced the behavior on my development machine as well - just building an image seems to create these extra directories for each layer of the image so I would run out of disk space really quick. Again, reverting to 17.06.0-ce seems to not have the problem so I'm going to stay there for now.
@mmanderson Thanks for reporting. Taking a look at changes in the AUFS driver.
@mmanderson Do you have any containers in the `Dead` state in `docker ps -a`?
All of my docker build servers are running out of space.
I have upgraded within the last week or so to Docker version 17.06.1-ce, build 874a737. I believe that nothing else has changed and that this issue either emerged or manifested as part of the upgrade process. The aufs diff directory is massive and I already pruned all images and dangling volumes.
[issue-22207.txt](https://github.com/moby/moby/files/1240116/issue-22207.txt)
@cpuguy83 No containers in any state. Here is what I just barely did to demonstrate this with 17.06.01-ce:
1. Started with a fresh install of docker 17.06.01-ce on Ubuntu 16.04.03 LTS (i.e. docker not installed and no /var/lib/docker directory). After install verified an empty /var/lib/docker/aufs/diff directory.
2. Ran a docker build with a fairly simple dockerfile based on ubuntu:latest - all it does is pull statsd_exporter from github and extract it into /usr/bin (see attached file).
3. After running the build run `docker ps -a` to show no containers in any state. There are several `*-remaining` folder in the `/var/lib/docker/aufs/diff` folder.
4. Run `docker system df` to verify images, container, and volumes. Result is
```
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 2 0 132.7MB 132.7MB (100%)
Containers 0 0 0B 0B
Local Volumes 0 0 0B 0B
```
6. Running `du -sch /var/lib/docker/*/` shows 152M for `/var/lib/docker/aufs/`
7. Run `docker rmi $(docker images -q)` to remove the built image layers. Running `docker system df` after this shows all zeros. Running `du -sch /var/lib/docker/*/` shows 152M for `/var/lib/docker/aufs/` and there are `*-remaining` folders for all of the folders that didn't have them before along with the existing `*-remaining` folders that are still there.
@erikh is this the issue you are experiencing?
@cpuguy83 After uninstalling 17.06.01-ce, removing the /var/lib/docker directory and installing 17.06.0-ce I try to run the same build. The build fails because of the `ADD from remote URL's` bug that was fixed in 17.06.01. However I don't get any `*-remaining` directories for the steps that do complete and after cleaning up everything with `docker system prune` and `docker rmi $(docker image -q)` the `/var/lib/docker/aufs/diff` directory is again empty and the space is freed.
Thanks all, this is a regression in 17.06.1...
PR to fix is here: https://github.com/moby/moby/pull/34587
awesome, thanks for the quick patch @cpuguy83! /cc @erikh
@rogaha! yes, thanks to you and @cpuguy83!
Thank you so much @Karreg for your [excellent script](https://github.com/moby/moby/issues/22207#issuecomment-322707352). After getting rid of all the old ophaned diffs and freeing huge amounts of lost disk space again we are using it now regularily to clean our VMs before installing new docker images. Great help and an almost perfect workaround for this issue now. @TP75
Looks like Docker, Inc. have some contracts with computer data storage manufacturers.
@Karreg's script worked fine for me and I freed all the space in the diffs directory.
Having the same issue.
Docker Host Details
root@UbuntuCont:~# docker info
Containers: 3
Running: 0
Paused: 0
Stopped: 3
Images: 4
Server Version: 17.06.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 14
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-93-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.358GiB
Name: UbuntuCont
ID: QQA5:DC5S:C2FL:LCC6:XY6E:V3FR:TRW3:VMOQ:QQKD:AP2M:H3JA:I6VX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
root@UbuntuCont:/var/lib/docker/aufs/diff# ls
031c85352fe85f07fede77dee0ac9dc2c7723177a819e72c534e1399208c95fa
09d53040e7e6798b5987ea76fe4f84f0906785b94a392a72e8e41a66cd9f242d
09d53040e7e6798b5987ea76fe4f84f0906785b94a392a72e8e41a66cd9f242d-init
0fb1ffc90969e9706801e2a18870f3ecd857a58f1094fbb968b3fa873e4cf2e4
10549179bd21a9c7af018d4ef305bb9196413b9662fce333b607104c40f38781
10d86a48e03cabf9af2c765dc84824809f24674ac339e4b9ffe572f50bd26b9c-init-removing
10d86a48e03cabf9af2c765dc84824809f24674ac339e4b9ffe572f50bd26b9c-removing
2e226946e8e6c2b3613de2afcff4cbb9890b6d9bd365fdda121a51ae96ec5606
2e226946e8e6c2b3613de2afcff4cbb9890b6d9bd365fdda121a51ae96ec5606-init
3601f6953132f557df8b52e03016db406168d3d6511d7ff5c08a90925ea288da-init-removing
3601f6953132f557df8b52e03016db406168d3d6511d7ff5c08a90925ea288da-removing
4b29141243aea4e70472f25a34a91267ab19c15071862c53e903b99740603d4c-init-removing
4b29141243aea4e70472f25a34a91267ab19c15071862c53e903b99740603d4c-removing
520e3fcf82e0fbbb48236dd99b6dee4c5bb9073d768511040c414f205c787dc5-init-removing
520e3fcf82e0fbbb48236dd99b6dee4c5bb9073d768511040c414f205c787dc5-removing
59cbb25a4858e7d3eb9146d64ff7602c9abc68509b8f2ccfe3be76681481904f
5d1c661b452efce22fe4e109fad7a672e755c64f538375fda21c23d49e2590f6
605893aba54feee92830d56b6ef1105a4d2166e71bd3b73a584b2afc83319591
63bd53412210f492d72999f9263a290dfee18310aa0494cb92e0d926d423e281-init-removing
63bd53412210f492d72999f9263a290dfee18310aa0494cb92e0d926d423e281-removing
72146e759ab65c835e214e99a2037f4b475902fdbe550c46ea0d396fb5ab2779-init-removing
72146e759ab65c835e214e99a2037f4b475902fdbe550c46ea0d396fb5ab2779-removing
8147e0b06dcbce4aa7eb86ed74f4ee8301e5fe2ee73c3a80dcb230bd0ddfcc26-init-removing
8147e0b06dcbce4aa7eb86ed74f4ee8301e5fe2ee73c3a80dcb230bd0ddfcc26-removing
a72735551217bb1ad01b77dbdbb9b8effa9f41315b0c481f8d74b5606c50deb4
aa58f2000b9f7d1ed2a6b476740c292c3c716e1d4dc04b7718580a490bba5ee8
b552cb853e33a8c758cb664aec70e2c4e85eacff180f56cbfab988a8e10c0174-removing
cd80c351b81ed13c4b64d9dfdc20c84f6b01cbb3e26f560faf2b63dae12dec55-init-removing
cd80c351b81ed13c4b64d9dfdc20c84f6b01cbb3e26f560faf2b63dae12dec55-removing
fe903be376821b7afee38a016f9765136ecb096c59178156299acb9f629061a2
fe903be376821b7afee38a016f9765136ecb096c59178156299acb9f629061a2-init
@kasunsjc please read the posts just above yours.
I confirm upgrading to 17.06.2-ce solved this issue. I didn't have to manually the directories either (last time) after the upgrade.
17.06.2-ce _appears_ to have fixed this for me as well. No more `-removing` directories in there, got a decent amount of space back.
I'm assuming that the `-init` directories I have in `aufs/diff` are unrelated (some of them are pretty old). They are all small, though, so it hardly matters.
Updating to 17.07.0 solved the issue here too, not even `docker system prune --all -f` would remove the directories before but after upgrading they got autoremoved on reboot.
Confirming this issue was resolved on Ubuntu 16.04 with 17.06.2-ce. As soon as it was updated, the space cleared.空#### Challenge Name
https://www.freecodecamp.com/challenges/sift-through-text-with-regular-expressions
#### Issue Description
Unable to edit the code at line that says //Change this Line
Clicking by mouse or using arrow keys on keyboard don't work, unless you go and edit some other line
and return to the line.
Issue is seen in Firefox and Chrome on mac and not on safari.
#### Browser Information
- Browser Name, Version: Google Chrome, 49.0.2623.110 (64-bit), Mozilla Firefox 44.0.2.
- Operating System: Mac OS X 10.11.3
- Mobile, Desktop, or Tablet: Desktop
#### Your Code
```
No user code required.
```
#### Screenshot
I've uploaded a video of the issue to understand it more, as an image won't suffice here.
https://youtu.be/2ScYLKg3xpU
+Closing as duplicate of #7847. Happy coding!
True class: both
Text with highlighted words
net: Support the /etc/resolver DNS resolution configuration hierarchy on OS X
https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man5/resolver.5.html
OS X allows you to add TLD specific resolver configurations. Quite popular ones are /etc/resolver/vm for local virtual machines and /etc/resolver/dev for local development purposes.
https://golang.org/src/net/dnsclient_unix.go#L231
Go seems to be hardcoded to only take /etc/resolv.conf into account on Unix platforms.
I don't think Go-native DNS resolving mechanism is used on Mac.
https://golang.org/src/net/dnsclient_unix.go#L231 is not executed if I run
``` go
addrs, err := net.LookupHost("google.com")
```
on my Mac.
If I enable debugging (`GODEBUG=netdns=2 go run test.go`), the following is printed:
```
go package net: using cgo DNS resolver
go package net: hostLookupOrder(google.com) = cgo
```
which means that OS-native DNS resolving is used.
Can you supply an exact configuration file, Go code, actual and expected output?
@nodirt This is for a binary with cgo off.
If cgo is disabled then the pure go DNS resolver will be used. If you want
to use the Mac DNS resolver, plese build with cgo.
On Mon, 7 Sep 2015 07:47 Jonathan Rudenberg notifications@github.com
wrote:
| @nodirt https://github.com/nodirt This is for a binary with cgo off.
|
| —
| Reply to this email directly or view it on GitHub
| https://github.com/golang/go/issues/12524#issuecomment-138128691.
Shouldn't be a problem since this is needed only on a dev machine.
On Sun, Sep 6, 2015 at 4:06 PM Dave Cheney notifications@github.com wrote:
| If cgo is disabled then the pure go DNS resolver will be used. If you want
| to use the Mac DNS resolver, plese build with cgo.
|
| On Mon, 7 Sep 2015 07:47 Jonathan Rudenberg notifications@github.com
| wrote:
|
| | @nodirt https://github.com/nodirt This is for a binary with cgo off.
| |
| | —
| | Reply to this email directly or view it on GitHub
| | https://github.com/golang/go/issues/12524#issuecomment-138128691.
|
| —
| Reply to this email directly or view it on GitHub
| https://github.com/golang/go/issues/12524#issuecomment-138134749.
In this specific case, @Rotonen was using the Flynn binary that we distribute as a compiled artifact, it is compiled without cgo to ease cross-compilation. Just because the user is a developer doesn't mean that they are a Go developer or want to compile the binary for themselves. The only question here is if this feature is out of scope for the pure-Go resolver.
cross compilation with cgo-enabled net package is not that hard.
You can reuse the package contained in binary distribution and
force internal linking.
I don't see anything wrong with supporting the OS X /etc/resolver directory. That said, my understanding is that the Go DNS resolver does not work well on most OS X machines. That is why it is disabled by default.
This would be great in all platforms anyway. Is there any disadvantage from supporting this behaviour? It seems that it'd neatly resolve the need to install and configure dnsmasq to provide the simple function of having different resolvers for different TLDs.
i know this issue is quite old but has there been any traction on this?
any resolution?
Any updates would be posted here. No updates have been posted here.
See `resolver(5)`. Just reading the files out of /etc/resolver/* will miss out on other mechanisms for configuring the same thing, for example configuration profiles or IKE attributes.
Just stumbled upon this today while attempting to use coredns as a dns proxy for local development. It's a real bummer to discover how naive our support for os x is.
We've generally assumed people use cgo on Darwin, so this bug has never been a priority.
I do admit that practically means that Darwin binaries need to be built on Darwin, which is difficult for people wanting to cross-compile for a dozen platforms as part of their release process.
Perhaps on Darwin without cgo we could just shell out to a program to do DNS resolution (e.g. host, dig, nslookup?). At least `nslookup` has an interactive mode that would permit re-using a child process for multiple lookups, if that proves necessary for performance.
I think reality is most command-line utilities will compile for two platforms: Linux and OS X, and the OS X build will always have cgo disabled. Some subset of the OS X users are using VPN, expect .local names to resolve, or have some other situation where hostname resolution is more than "just query this one DNS server always". Some subset of those users will actually open an issue with the tool, and of those even a smaller subset identify go as the problem and raise an issue here.
So I think you underestimate the impact of the problem.
Shelling out to `nslookup` will not fix it. The problem is "doing a DNS query" is not the same thing as "resolving a hostname". Resolving a hostname involves more, such as:
- /etc/hosts
- [RFC6762](https://tools.ietf.org/html/rfc6762) `.local` names
- Other hostname resolution protocols, such as [NIS](https://www.thkukuk.de/nis/) or [LDAP](https://serverfault.com/questions/166981/how-to-configure-ldap-to-resolve-host-names), if configured
- Honoring the domain search path, if configured
- If DNS is to be used, determining which server to use.
Tools like `host`, `nslookup`, and `dig` do DNS queries by design, not resolve hostnames. This is equally true on Linux as well as OS X. Unfortunately somehow OS X has acquired some lore about having "two DNS systems", which is simply false. Or at least it was false, until go command-line utilities gained popularity.
If you do want to shell out to a command to perform host resolution, the correct command on OS X is `dscacheutil -q host -a name $hostname`. This is analogous to `getent hosts $hostname` on Linux.
Another path is to make the go resolver's behavior more consistent with the OS X system resolver. This begins with obtaining resolver configuration from [SystemConfiguration.framework](https://developer.apple.com/documentation/systemconfiguration) or `scutil --dns`, _not_ `/etc/resolv.conf`.
`dscacheutil` sounds good. I was thinking of `lookupd` when I wrote the comment above but my local machine didn't have `lookupd` so I omitted it. Now I see that `dscacheutil` replaced `lookupd`.
I don't think we want to get into the business of reimplementing Darwin's name resolution.
@randall77, since you're having fun with macOS lately, any thoughts here? Could we have non-cgo binaries still call into the macOS name resolution code somehow with some assembly/linker goo?
Let's see if we can use the libSystem bindings directly even when cgo is ostensibly disabled.
| expect .local names to resolve
I actually expect `.local` names to resolve on all platforms per mDNS anyway, if the target responds to the broadcast appropriately.
@bitglue is correct. I think a lot of people are going to file issues against a tool and not raise issues to the Go project. A good example of this is Homebrew. They recently removed support for options in their install which now means people can't install packages written in Go, like Hashicorp's Vault with cgo support. We used to be able to do 'brew install vault --with-dyanmic' to enable cgo support to get correct DNS resolution, but now that is removed and we're stuck with having to hack their install script to get Vault compiled with cgo. It would be nice to see Go's native resolver work in a less naive fashion so we don't need to worry about this issue anymore.
See https://github.com/Homebrew/homebrew-core/issues/33507 for reference.
I would chime in and venture that the root of this issue might be that the `net` package treats all Unix systems the same. Perhaps there should be a stubbed out variant for MacOS to deal with it's `configd` based resolution?
This issue, as has been noted, will affect every binary not compiled with cgo when users are using VPNs, which would seem to be a common use case.
@rsc Can you provide some detail on how we might be able to call libSystem bindings without cgo?
@grantseltzer The current runtime package is full of examples of calling into libSystem. See runtime/sys_darwin.go.
I'm taking a stab at this, I have a branch on my github fork here: https://github.com/grantseltzer/go but could use some help
The function call i'm looking for is [res_search](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/res_query.3.html) which is in libresolv (`/usr/lib/libresolv.9.dylib`)
I have the cgo_import_dynamic directive:
`//go:cgo_import_dynamic libresolv_res_search res_search "/usr/lib/libresolv.9.dylib"`
The Go function that makes the libcCall call and trampoline (`sys_darwin.go`):
```go
//go:nosplit
//go:cgo_unsafe_args
func Res_search(name *byte, class int32, rtype int32, answer *byte, anslen int32) int32 {
return libcCall(unsafe.Pointer(funcPC(res_search_trampoline)), unsafe.Pointer(|name))
}
func res_search_trampoline()
```
and defined the amd64 assembly routine (`sys_darwin_amd64.s`):
```asm
TEXT runtime·res_search_trampoline(SB),NOSPLIT,$0
PUSHQ BP
MOVQ SP, BP
MOVL 0(DI), SI // arg 1 name
MOVQ 8(DI), DX // arg 2 class
MOVQ 12(DI), CX // arg 3 type
MOVQ 16(DI), R8 // arg 4 answer
MOVQ 24(DI), R9 // arg 5 anslen
CALL libresolv_res_search(SB)
POPQ BP
RET
```
When testing the function (which is exported just for testing), I get a return code of `-1` and no response in buffer:
```go
func main() {
name := "google.com"
var nameAddr = name[0]
var buffer = [512]byte{}
x := runtime.Res_search(|nameAddr, 255,
255, |buffer[0], 512)
fmt.Println("res_search return code:", x)
fmt.Printf("Buffer: %s\n", buffer)
}
```
Anything glaring that i'm missing? Perhaps my datatypes or stack offset sizes.
Most importantly, can someone link me to documentation on how to debug the code at this level?
EDIT:
more testing/version information:
uname -a
```
Darwin Grant-SelzterRichman 17.7.0 Darwin Kernel Version 17.7.0: Thu Dec 20 21:47:19 PST 2018; root:xnu-4570.71.22~1/RELEASE_X86_64 x86_64
```
```
go version go1.11.5 darwin/amd64
```
CC @randall77
I believe I was misusing MOVQ vs MOVL (now potentially fixed to this):
```asm
TEXT runtime·res_search_trampoline(SB),NOSPLIT,$0
PUSHQ BP
MOVQ SP, BP
MOVQ 0(DI), DI // arg 1 name
MOVL 8(DI), SI // arg 2 class
MOVL 12(DI), DX // arg 3 type
MOVQ 16(DI), CX // arg 4 answer
MOVL 24(DI), R8 // arg 5 anslen
CALL libresolv_res_search(SB)
POPQ BP
RET
```
Still not there yet though.
I'm stepping through with delve and my hunch is that RDI has not been properly initialized when entering the res_search_trampoline in `sys_darwin_amd64.s`
When moving from offsets off DI to the respective arg registers the program appears to be blowing away the destination registers instead (pictured below):
Another thing that's confusing me is that when I step into `Res_search` (the go function that makes the call to `libcCall`) my arguments are unreadable:
Anyone have a hunch of why this call isn't working or have advice on debugging?
Update:
I am getting DNS records using the libresolv `res_search` binding with cgo disabled :D!
Working to confirm that this actually honors the `/etc/resolver` files, not sure if it is at the moment.
Would still love to hear an explanation for this, but the way I got it working was by changing the order of the arguments being loaded to the order of them listed in the dlv screenshot above:
```asm
TEXT runtime·res_search_trampoline(SB),NOSPLIT,$0
PUSHQ BP
MOVQ SP, BP
MOVL (DI), R8 // arg 5 anslen
MOVQ 16(DI), CX // arg 4 answer
MOVL 8(DI), SI // arg 2 class
MOVQ 0(DI), DI // arg 1 name
MOVL 12(DI), DX // arg 3 type
CALL libresolv_res_search(SB)
POPQ BP
RET
```
Current update: Calling this routine does in fact honor `/etc/resolver/` files. ~~I'm currently trying to figure out an issue where the specified query 'type' is not being honored and only AAAA queries are sent.~~
My questions for once I fix that and prepare it for a CL:
1) Should this routine be defined for all of i386, x86_64, ARM, and ARM64?
2) What testing mechanisms exist for code at this level beyond manually?
3) Should the cgo bindings exist in runtime or are they appropriate for the net package?
Opened #30686
Change https://golang.org/cl/166297 mentions this issue: `net: Use libSystem bindings for DNS resolution on macos if CGO is unavailable`
If we want to accommodate several DNS stub resolver implementations, typically it would be as follows:
- well-cooked external getaddrinfo based one; currently enabled by netdns=cgo,
- half-baked external resolver library, res_xxx, based one,
- from scratch; currently enabled by netdns=go.
However, I'm still not sure we really need to hold all of the implementations in the package net. Is there any specific reason not making a new API that accepts external stub resolver implementations? Once we open up the API, we are also able to use the API for upcoming fancy technologies such as DoH (DNS over HTTPS).
```
TEXT runtime·res_search_trampoline(SB),NOSPLIT,$0
PUSHQ BP
MOVQ SP, BP
MOVL (DI), R8 // arg 5 anslen
MOVQ 16(DI), CX // arg 4 answer
MOVL 8(DI), SI // arg 2 class
MOVQ 0(DI), DI // arg 1 name
MOVL 12(DI), DX // arg 3 type
CALL libresolv_res_search(SB)
POPQ BP
RET
```
The last `MOVL` is using a `DI` value that just got clobbered in the previous instruction. You have to load `DI` last.
The manpage is unclear about what the return value of res_search is. You might need to call `libc_error` if the return value is |0 to get an actual error code. See mmap for an example.
Debugging this stuff is hard generally. Sorry about that. It does seem that you're making progress though.
By the way, if Darwin supports `res_nsearch` and friends, we should probably use them, as they are thread-safe. `res_search` and `res_nsearch` normally return the length of the response and I assume the same is true on Darwin.
@randall77 Ah that makes a lot of sense, thank you! I pushed changes including the error checks (they return size of response, unless error which is -1)
@ianlancetaylor I have been working on this today, as well as changing the GODEBUG/CGO set logic discussed on gerrit.
`res_nsearch` is supported.
In order to use `res_nsearch` we would have to use `res_ninit`. I don't know whether `res_search` would also work OK, but it's troubling that it's not considered to be thread-safe on GNU/Linux. I don't know about Darwin. I don't know when the global variable is modified.
But I guess that to use `res_ninit` and `res_nsearch` we would need to at least know the size of `res_state`. Probably the best approach would be to double-check that on Darwin `res_state` is |= 512 bytes, as I expect it is, and then use `[64]uint64`.
Change https://golang.org/cl/180842 mentions this issue: `net: fix non-cgo macOS resolver code`
Given that we already use the C library with cgo-based macOS builds (the default) and that we in fact prefer cgoLookupHost to doing it ourselves, it seems like Go should support /etc/resolver just fine out of the box.
CL 166297 (f6b42a5) added some code for the non-cgo builds, but (1) it doesn't work and (2) it's unclear that the non-cgo builds really need attention to this corner case.
I sent CL 180843 to revert the recent changes, but I am inclined to leave this bug closed, since again the cgo path should be handling /etc/resolver just fine.
It seems odd that a deprecated mechanism on macOS would be the best alternative. Perhaps a native builder for Darwin would fix the upstream issues?
| it's unclear that the non-cgo builds really need attention to this corner case.
This is an actual issue for us, as it presumably is for the original reporter as well as others who've chimed in on this thread.
| This is an actual issue for us, as it presumably is for the original reporter
I've circumvented this since 2015 by not using [Flynn](https://flynn.io/), and thus not needing the functionality in a non-cgo Go on Darwin. That decision had nothing to do with this issue: I circumvented by beefing up my machine-internal infrastructure stack so I did not have to rely on the macOS `/etc/resolver/*` mechanism for rolling my own private TLD.
I've not encountered any software, Go or otherwise, since then, which would not work on Darwin with my machine-local VM cluster and networking setup using the `/etc/resolver/*` mechanism. I still use the machine-local infra stack for evaluating new infrastructure stacks from time to time. The circumstances for ending up in this corner are fairly specific - when putting on the systems consultant hat from time to time, I am a lone wolf for whom everything needs to work laptop-internally for being able to go and showcase things trivially.
@cespare perhaps the real solution for you would be to bake the contextual resolver dynamicity into your corporate networking infrastructure and not try to do a full OSI layer cake in-machine. Just roll your own network-internal TLD root.
Or figure out cross compilation - it is less scary than it sounds like.
@rsc While it's true that that CGO is enabled by default in the compiler, it is consistently disabled by the maintainers of flagship Golang applications:
https://github.com/kubernetes/kubernetes/blob/v1.14.2/hack/lib/golang.sh#L377-L410
https://github.com/hashicorp/consul/blob/v1.5.1/build-support/functions/20-build.sh#L456
https://github.com/hashicorp/terraform/blob/v0.12.1/scripts/build.sh#L43
https://github.com/hashicorp/vault/blob/v1.1.3/Makefile#L18
This means, for all these tools, when run on an OS X machine, their DNS resolving is broken beyond the happy path of vanilla DNS - which from my personal experience, while admittedly short, has never been the case in an enterprise workplace, primarily due to VPNs.
It also means that whenever someone discovers that DNS is broken in any Golang tool they use, they'll eventually discover this thread, where we effectively told them to go pound sand, and take the issue up with the maintainer(s) of their tool as to why it was too difficult for them to build and release their tool with the defaults turned on.
As to what the challenges are with leaving CGO enabled, I would encourage you to take a look at the issues @caarlos0 has had while trying to support CGO in the fabulous build tool GoReleaser:
https://github.com/goreleaser/goreleaser/issues/708
Additionally, the folks at Homebrew, a commonly used package manager for OS X, are making it increasingly more difficult to even support the option for maintainers to support installation flags, further increasing the issue:
https://github.com/Homebrew/homebrew-core/issues/33507
Ultimately, it's up to you @rsc and the example you want to lead with. It's without a doubt beyond obnoxious that so many systems have failed to get to the point where we have to even consider solving the problem in the language - nevertheless, here we are looking for a hero.
Add our non-corner case problem with this https://github.com/vapor-ware/ksync/issues/260
@rsc Between the comments above, ones on the original issue, and through speaking to people in person and on slack I know a lot of people/orgs could really use your fixed change set. Kubectl, helm, vpn services, hashicorp tools, and many many others are affected by the lack of this feature.
What would you need to see to overturn your decision?空Build macos `oc` binary with CGO_ENABLED=1 so that it honours /etc/resolver?
|!--
Note that this is the issue tracker for OpenShift Origin.
OpenShift Installer issues should be reported at https://github.com/openshift/openshift-ansible/issues
Management Console issues are collected at https://github.com/openshift/origin-web-console/issues
Documentation issues are better reported at https://github.com/openshift/openshift-docs/issues
--|
##### Version
```
$ oc version
oc v4.0.0-0.171.0
kubernetes v1.12.4+a532756e37
```
##### Steps To Reproduce
1. Create a cluster locally which doesn't have public dns lookup (you can use crc for it).
2. Don't append the dns details to `/etc/resolve.conf` but use `/etc/resolver/|whatever_dns|` , in our case we use `/etc/resolver/testing`.
3. Try to open the web console on the browser which works and it honour the resolver file.
4. Try to use oc cli to login to cluster, it doesn't work until you add that dns to `/etc/resolv.conf`
##### Current Result
User need to add the dns details to `/etc/resolv.conf`, where most of the app on the Mac does honour the resolver file and works.
##### Expected Result
Openshift client binary should also honour the `resolver` directory files.
##### Additional Information[try to run `$ oc adm diagnostics` (or `oadm diagnostics`) command if possible]
[if you are reporting issue related to builds, provide build logs with `BUILD_LOGLEVEL=5`]
[consider attaching output of the `$ oc get all -o json -n |namespace|` command to the issue]
[visit https://docs.openshift.org/latest/welcome/index.html]
True class: both
Text with highlighted words
helm upgrade fails with spec.clusterIP: Invalid value: "": field is immutable
When issue helm upgrade, it shows errors like, ("my-service" change from "clusterIP: None" to "type: LoadBalancer" without field clusterIP)
```
Error: UPGRADE FAILED: Service "my-service" is invalid: spec.clusterIP: Invalid value: "": field is immutable
```
However, all other pods with new version are still going to be restarted, except that "my-service" Type does not change to new type "LoadBalancer"
I understand that why upgrade failed because helm does not support changing on some certain fields. But why helm still upgrades other services/pods by restarting it. Should helm does nothing if there is any error during the upgrade? I excepted helm to treat the whole set of services as a package to either upgrade all or none, but seems my expectation might be wrong.
And if we ever end up in such situation, then what we should to get out the situation? like how to upgrade "my-service" to have new type?
And if I use --dry-run option, helm does not show any errors.
Is this consider a bug or expected, i.e. upgrade throws some error but some service still gets upgraded.
|!-- If you need help or think you have found a bug, please help us with your issue by entering the following information (otherwise you can delete this text): --|
Output of `helm version`:
```
Client: |version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Server: |version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
```
Output of `kubectl version`:
```
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.27", GitCommit:"145f9e21a4515947d6fb10819e5a336aff1b6959", GitTreeState:"clean", BuildDate:"2020-02-21T18:01:40Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}
```
Cloud Provider/Platform (AKS, GKE, Minikube etc.):
GKE and Minkube
Not enough information has been provided to reproduce. Please tell us how to create a reproducible chart, and which Helm commands you used.
Hi, here are the reproduce steps
Having two services yaml file as below.
nginx.yaml
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
```
prometheus.yaml
```
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: prometheus
spec:
template:
metadata:
labels:
app: prometheus
spec:
containers:
- image: prom/prometheus
name: prometheus
ports:
- containerPort: 9090 imagePullPolicy: Always
hostname: prometheus
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: prometheus
spec:
selector:
app: prometheus
clusterIP: None
ports:
- name: headless
port: 9090
targetPort: 0
```
Then put there two files in helm1/templates/ then install. It shows prometheus service uses clusterIP and nginx version is 1.14.2
```
# helm upgrade --install test helm1
Release "test" does not exist. Installing it now.
NAME: test
LAST DEPLOYED: Tue Apr 21 20:42:55 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 |none| 443/TCP 35d
prometheus ClusterIP None |none| 9090/TCP 7s
# kubectl describe deployment nginx |grep Image
Image: nginx:1.14.2
```
Now update the section for nginx.yaml to new version 1.16
```
image: nginx:1.16
```
and prometheus.yaml by changing it to LoadBalancer.
```
spec:
selector:
app: prometheus
ports:
- name: "9090"
port: 9090
protocol: TCP
targetPort: 9090
type: LoadBalancer
```
Now put them as helm2 and do the upgrade. Then you can see the upgrade throw some errors, but the nginx service goes through, by upgrade to a new version, but the prometheus is not upgraded as it is still using Cluster IP.
```
# helm upgrade --install test helm2
Error: UPGRADE FAILED: cannot patch "prometheus" with kind Service: Service "prometheus" is invalid: spec.clusterIP: Invalid value: "": field is immutable
# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 |none| 443/TCP 35d
prometheus ClusterIP None |none| 9090/TCP 5m34s
# kubectl describe deployment nginx |grep Image
Image: nginx:1.16
```
helm list shows
```
# helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
test default 2 2020-04-21 20:48:20.133644429 -0700 PDT failed
```
helm history
```
# helm history test
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Tue Apr 21 20:42:55 2020 deployed helm-helm 1.0.0.6 Install complete 2 Tue Apr 21 20:48:20 2020 failed helm-helm 1.0.0.6 Upgrade "test" failed: cannot patch "prometheus" with kind Service: Service "prometheus" is invalid: spec.clusterIP: Invalid value: "": field is immutable
```
We have the same behavior with v3.2.0, downgrading to v3.1.3 is our temporary fix
I've got a lot of this with my Helm 2 -| 3 migration. When trying to upgrade the converted Releases for the first time I get a lot of this. This is for Nginx Ingress, Prometheus Operator, Graylog and Jaeger charts so far. Most of them I'm content with just deleting the services and letting Helm recreate them but for Nginx Ingress this isn't an option...
Just found this https://github.com/helm/helm/issues/6378#issuecomment-557746499 which explains the problem in my case.
Closing as a duplicate of #6378. @cablespaghetti found the deeper explanation for this behaviour, which is described in great detail.
Let us know if that does not work for you.
@GaramNick why would downgrading fix this for you? Can you elaborate more on “what” was fixed by downgrading?
@bacongobbler While you're here. Is there any way to fix this situation without deleting the release and re-deploying? I can't seem a way to do that under helm 2 or 3. I want to hack the existing release data so Helm thinks the clusterIP has always been omitted and so no patch is necessary.
Have you tried `kubectl edit`?
We have the same issue and downgrading to `3.1.3` fixed it also for us. My guess is that it has to do with the new logic in https://github.com/helm/helm/pull/7649/commits/d829343c1514db17bee7a90624d06cdfbffde963 considering this a `Create` and not an update thus trying to set empty IP and not reusing the populated one
Interesting find. thank you for investigating.
@jlegrone any chance you might have time to look into this?
@bacongobbler Our CI/CD pipeline uses Helm to update our application that includes a Service with type ClusterIP. The command:
```bash
helm upgrade --install --force \
--wait \
--set image.repository="$CI_REGISTRY_IMAGE" \
--set image.tag="$CI_COMMIT_REF_NAME-$CI_COMMIT_SHA" \
--set image.pullPolicy=IfNotPresent \
--namespace="$KUBE_NAMESPACE" \
"$APP_NAME" \
./path/to/charts/
```
On v3.2.0 this command fails with `Service "service-name" is invalid: spec.clusterIP: Invalid value: "": field is immutable`
On v3.1.3 this works fine.
Let me know if you like to have more info.
Same here. We had the following service.yaml working fine with helm2 for many many months.
After migration, the helm 3.2 `helm upgrade` command failed with the save error as above. Downgrading to 3.1.3 resolved it.
```
apiVersion: v1
kind: Service
metadata:
name: {{ .Values.global.name }}
namespace: {{ index .Values.global.namespace .Values.global.env }}
labels:
microservice: {{ .Values.global.name }}
spec:
type: ClusterIP
ports:
- port: 8080
selector:
microservice: {{ .Values.global.name }}
```
| We have the same issue and downgrading to 3.1.3 fixed it also for us. My guess is that it has to do with the new logic in d829343 considering this a Create and not an update thus trying to set empty IP and not reusing the populated one
@n1koo Can you explain why you think this is the code causing the issue? As this is the install and not upgrade code, and also the [code in 3.1](https://github.com/helm/helm/blob/release-3.1/pkg/action/install.go#L293) is a ``create` and it works.
I am reviewing the issue with @adamreese , and we _think_ it is the patch that @n1koo identified. The Create method will bypass the normal 3-way diff on the Service, which will result in the service's clusterIP being set to "" instead of the value populated by Kubernetes. As a result, the manifest sent to the API server _appears_ to be resetting the cluster IP, which is illegal on a service (and definitely not what the user intended).
We're still looking into this and I will update if we learn more.
So https://github.com/helm/helm/issues/6378#issuecomment-557746499 is correct. Please read that before continuing on with this issue. If `clusterIP: ""` is set, Kubernetes will assign an IP. On the next Helm upgrade, if `clusterIP:""` again, it will give the error above, because it appears _to Kubernetes_ that you are trying to reset the IP. (Yes, Kubernetes modifies the `spec:` section of a service!)
When the `Create` method bypasses the 3-way diff, it sets `clusterIP: ""` instead of setting it to the IP address assigned by Kubernetes.
To reproduce:
```
$ helm create issue7956
$ # edit issue7956/templates/service.yaml and add `clusterIP: ""` under `spec:`
$ helm upgrade --install issue7956 issue7956
...
$ helm upgrade issue7956 issue7956
Error: UPGRADE FAILED: cannot patch "issue-issue7956" with kind Service: Service "issue-issue7956" is invalid: spec.clusterIP: Invalid value: "": field is immutable
```
The second time you run the upgrade, it will fail.
I cannot reproduce @IdanAdar 's case on `master`.
@GaramNick there is not enough info about the service you are using for us to reproduce your error.
My situation:
`version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}`
also tested w/
`version.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}`
given the following service template:
```
apiVersion: v1
kind: Service
metadata:
name: {{ include "app.fullname" . }}
labels:
{{- include "app.labels" . | nindent 4 }}
annotations:
getambassador.io/config: |
---
apiVersion: ambassador/v1
kind: Mapping
name: {{ include "app.fullname" . }}_mapping
prefix: /{{ include "app.fullname" . }}
host: "^{{ include "app.fullname" . }}.*"
host_regex: true
service: {{ include "app.fullname" . }}.{{ .Release.Namespace }}
rewrite: ""
timeout_ms: 60000
bypass_auth: true
cors:
origins: "*"
methods: POST, GET, OPTIONS
headers:
- Content-Type
- Authorization
- x-client-id
- x-client-secret
- x-client-trace-id
- x-flow-proto
---
apiVersion: ambassador/v1
kind: Mapping
name: {{ include "app.fullname" . }}_swagger_mapping
ambassador_id: corp
prefix: /swagger
host: "^{{ include "app.fullname" . }}.corp.*"
host_regex: true
service: {{ include "app.fullname" . }}.{{ .Release.Namespace }}
rewrite: ""
bypass_auth: true
cors:
origins: "*"
methods: POST, GET, OPTIONS
headers:
- Content-Type
- x-client-id
- x-client-secret
- Authorization
- x-flow-proto
namespace: {{ .Release.Namespace }}
spec:
type: {{ .Values.service.type }}
selector:
{{- include "app.selectorLabels" . | nindent 4 }}
ports:
- port: {{ .Values.service.port }}
name: http-rest-hub
targetPort: http-rest
- port: {{ .Values.service.healthPort }}
name: http-health
targetPort : http-health
```
which results in the following after `upgrade --install`:
```
apiVersion: v1
kind: Service
metadata:
annotations:
getambassador.io/config: |
---
apiVersion: ambassador/v1
kind: Mapping
name: hub-alt-bor_mapping
prefix: /hub-alt-bor
host: "^hub-alt-bor.*"
host_regex: true
service: hub-alt-bor.brett
rewrite: ""
timeout_ms: 60000
bypass_auth: true
cors:
origins: "*"
methods: POST, GET, OPTIONS
headers:
- Content-Type
- Authorization
- x-client-id
- x-client-secret
- x-client-trace-id
- x-flow-proto
---
apiVersion: ambassador/v1
kind: Mapping
name: hub-alt-bor_swagger_mapping
ambassador_id: corp
prefix: /swagger
host: "^hub-alt-bor.corp.*"
host_regex: true
service: hub-alt-bor.brett
rewrite: ""
bypass_auth: true
cors:
origins: "*"
methods: POST, GET, OPTIONS
headers:
- Content-Type
- x-client-id
- x-client-secret
- Authorization
- x-flow-proto
meta.helm.sh/release-name: alt-bor
meta.helm.sh/release-namespace: brett
creationTimestamp: ...
labels:
app: hub
app.kubernetes.io/instance: alt-bor
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: hub
app.kubernetes.io/version: v1.6.0-rc.26
deploy.xevo.com/stackname: bor-v0.1-test
helm.sh/chart: hub-0.0.4
owner: gateway
ownerSlack: TODOunknown
name: hub-alt-bor
namespace: brett
resourceVersion: ...
selfLink: ...
uid: ...
spec:
clusterIP: 172.20.147.13
ports:
- name: http-rest-hub
port: 80
protocol: TCP
targetPort: http-rest
- name: http-health
port: 90
protocol: TCP
targetPort: http-health
selector:
app.kubernetes.io/instance: alt-bor
app.kubernetes.io/name: hub
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
```
If I then upload this exact same chart as version 0.0.5 and `upgrade --install` again I get the following:
`Error: UPGRADE FAILED: failed to replace object: Service "hub-alt-bor" is invalid: spec.clusterIP: Invalid value: "": field is immutable`
The only difference is the value of the `helm.sh/chart` label which now has a value of `hub-0.0.5`
This is a huge blocker.
| @GaramNick there is not enough info about the service you are using for us to reproduce your error.
@technosophos What do you need? Happy to provide more details!
Update! The update fails ONLY when using `helm upgrade --install` w/ `--force`. Less of a blocker now.
Oh! That is interesting. That should make the error easier to track down.
Hello @technosophos @bacongobbler we have the same 2 issues:
`version.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}`
1. Issue
We have `Service` template without `clusterIP` but kubernetes will assign `clusterIP` automatically:
```
apiVersion: v1
kind: Service
metadata:
name: {{ .Release.Name }}
labels:
app: {{ .Values.image.name }}
release: {{ .Release.Name }}
spec:
type: ClusterIP
ports:
- port: {{ .Values.service.port }}
targetPort: {{ .Values.service.port }}
protocol: TCP
name: http
selector:
app: {{ .Values.image.name }}
release: {{ .Release.Name }}
```
after migrate to helm 3 with `helm 2to3 convert` and try upgrade the same release `helm3 upgrade --install --force`:
```
failed to replace object: Service "dummy-stage" is invalid: spec.clusterIP: Invalid value: "": field is immutable
```
if i will do the same without `--force` -| `helm3 upgrade --install` works fine without error.
2. Issue
if I want change `spec.selector.matchLabels` in Deployment which are immutable field without `--force` I get error:
```
cannot patch "dummy-stage" with kind Deployment: Deployment.apps "dummy-stage" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/name":"web-nerf-dummy-app"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
```
if I will do the same with `--force` I get error:
```
failed to replace object: Deployment.apps "dummy-stage" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/name":"web-nerf-dummy-app"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
```
Is it possible implement the same behaviour for `--force` as in `helm 2` because we can without any error upgrade immutable filed ?
```
apiVersion: v1
kind: Service
metadata:
name: zipkin-proxy
namespace: monitoring
spec:
ports:
- port: 9411
targetPort: 9411
selector:
app: zipkin-proxy
type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: zipkin-proxy
namespace: monitoring
spec:
replicas: {{ .Values.zipkinProxy.replicaCount }}
template:
metadata:
labels:
app: zipkin-proxy
annotations:
prometheus.io/scrape: 'true'
spec:
containers:
- image: {{ .Values.image.repository }}/zipkin-proxy
name: zipkin-proxy
env:
- name: STORAGE_TYPE
value: stackdriver
```
`helm upgrade -i --debug --force --namespace monitoring zipkin-proxy --values ./values.yaml.tmp .
`
I have tried with removing the force option. I tried with v3.1.3, v3.2.0 as well as v3.2.1 still the same issue.
Stack trace
```
history.go:52: [debug] getting history for release zipkin-proxy
upgrade.go:84: [debug] preparing upgrade for zipkin-proxy
upgrade.go:92: [debug] performing update for zipkin-proxy
upgrade.go:234: [debug] creating upgraded release for zipkin-proxy
client.go:163: [debug] checking 2 resources for changes
client.go:195: [debug] error updating the resource "zipkin-proxy":
cannot patch "zipkin-proxy" with kind Service: Service "zipkin-proxy" is invalid: spec.clusterIP: Invalid value: "": field is immutable
client.go:403: [debug] Looks like there are no changes for Deployment "zipkin-proxy"
upgrade.go:293: [debug] warning: Upgrade "zipkin-proxy" failed: cannot patch "zipkin-proxy" with kind Service: Service "zipkin-proxy" is invalid: spec.clusterIP: Invalid value: "": field is immutable
Error: UPGRADE FAILED: cannot patch "zipkin-proxy" with kind Service: Service "zipkin-proxy" is invalid: spec.clusterIP: Invalid value: "": field is immutable
helm.go:75: [debug] cannot patch "zipkin-proxy" with kind Service: Service "zipkin-proxy" is invalid: spec.clusterIP: Invalid value: "": field is immutable
helm.sh/helm/v3/pkg/kube.(*Client).Update
/home/circleci/helm.sh/helm/pkg/kube/client.go:208
helm.sh/helm/v3/pkg/action.(*Upgrade).performUpgrade
/home/circleci/helm.sh/helm/pkg/action/upgrade.go:248
helm.sh/helm/v3/pkg/action.(*Upgrade).Run
/home/circleci/helm.sh/helm/pkg/action/upgrade.go:93
main.newUpgradeCmd.func1
/home/circleci/helm.sh/helm/cmd/helm/upgrade.go:137
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
main.main
/home/circleci/helm.sh/helm/cmd/helm/helm.go:74
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357
UPGRADE FAILED
main.newUpgradeCmd.func1
/home/circleci/helm.sh/helm/cmd/helm/upgrade.go:139
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
main.main
/home/circleci/helm.sh/helm/cmd/helm/helm.go:74
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357
```
I am having this issue when the Helm Chart version changes and having an existing deployment.
Using Helm v3.2.0
I can confirm that downgrading to 3.1.2 works.空spec.clusterIP: Invalid value: "": field is immutable
```
apiVersion: v1
kind: Service
metadata:
name: zipkin-proxy
namespace: monitoring
spec:
ports:
- port: 9411
targetPort: 9411
selector:
app: zipkin-proxy
type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: zipkin-proxy
namespace: monitoring
spec:
replicas: {{ .Values.zipkinProxy.replicaCount }}
template:
metadata:
labels:
app: zipkin-proxy
annotations:
prometheus.io/scrape: 'true'
spec:
containers:
- image: {{ .Values.image.repository }}/zipkin-proxy
name: zipkin-proxy
env:
- name: STORAGE_TYPE
value: stackdriver
```
`helm upgrade -i --debug --force --namespace monitoring zipkin-proxy --values ./values.yaml.tmp .
`
**I have tried with removing the force option. I tried with _**v3.1.3, v3.2.0 as well as v3.2.1**_ still the same issue.**
Stack trace
```
history.go:52: [debug] getting history for release zipkin-proxy
upgrade.go:84: [debug] preparing upgrade for zipkin-proxy
upgrade.go:92: [debug] performing update for zipkin-proxy
upgrade.go:234: [debug] creating upgraded release for zipkin-proxy
client.go:163: [debug] checking 2 resources for changes
client.go:195: [debug] error updating the resource "zipkin-proxy":
cannot patch "zipkin-proxy" with kind Service: Service "zipkin-proxy" is invalid: spec.clusterIP: Invalid value: "": field is immutable
client.go:403: [debug] Looks like there are no changes for Deployment "zipkin-proxy"
upgrade.go:293: [debug] warning: Upgrade "zipkin-proxy" failed: cannot patch "zipkin-proxy" with kind Service: Service "zipkin-proxy" is invalid: spec.clusterIP: Invalid value: "": field is immutable
Error: UPGRADE FAILED: cannot patch "zipkin-proxy" with kind Service: Service "zipkin-proxy" is invalid: spec.clusterIP: Invalid value: "": field is immutable
helm.go:75: [debug] cannot patch "zipkin-proxy" with kind Service: Service "zipkin-proxy" is invalid: spec.clusterIP: Invalid value: "": field is immutable
helm.sh/helm/v3/pkg/kube.(*Client).Update
/home/circleci/helm.sh/helm/pkg/kube/client.go:208
helm.sh/helm/v3/pkg/action.(*Upgrade).performUpgrade
/home/circleci/helm.sh/helm/pkg/action/upgrade.go:248
helm.sh/helm/v3/pkg/action.(*Upgrade).Run
/home/circleci/helm.sh/helm/pkg/action/upgrade.go:93
main.newUpgradeCmd.func1
/home/circleci/helm.sh/helm/cmd/helm/upgrade.go:137
github.com/spf13/cobra.(*Command).execute/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
main.main
/home/circleci/helm.sh/helm/cmd/helm/helm.go:74
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357
UPGRADE FAILED
main.newUpgradeCmd.func1
/home/circleci/helm.sh/helm/cmd/helm/upgrade.go:139
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
main.main
/home/circleci/helm.sh/helm/cmd/helm/helm.go:74
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357
```
Output of `helm version`:
```
version.BuildInfo{Version:"v3.1.3", GitCommit:"0a9a9a88e8afd6e77337a3e2ad744756e191429a", GitTreeState:"clean", GoVersion:"go1.13.10"}
```
Output of `kubectl version`:
```
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.36", GitCommit:"34a615f32e9a0c9e97cdb9f749adb392758349a6", GitTreeState:"clean", BuildDate:"2020-04-06T16:33:17Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}
```
Cloud Provider/Platform (AKS, GKE, Minikube etc.):
GKE
If I add the random ClusterIP from my pod CIDR range it works. But I have 50 chart modules can't do for all.
Hi @azarudeena. Have you looked at the explanation provided in #6378? This looks identical to the symptoms provided in that ticket. If so this should probably be closed as a duplicate issue.
True class: both
Text with highlighted words
AKS API server support of standard load balancer (not user deployed services)
**What happened**:
When using the [Kubernetes Python API](https://github.com/kubernetes-client/python), after approximately 5 minutes of inactivity, the connection stops working. The packets are seemingly dropped on the other end.
In the case of the Python API, this causes the request to block for 15 minutes until the Linux Kernel forces the socket closed with "Errno 110 Connection timed out". The other side is closing the connection but not sending a TCP `RST`, leaving the socket open on the client.
Under the hood the Python API uses a connection pool that reuses the TCP Connections using the standard HTTP Keep-Alive mechanism.
This causes issues with the out of the box configuration in applications such as [Airflow](https://github.com/apache/airflow) that use the Kubernetes Python API.
I would assume other Kubernetes client libraries would have a similar issue, but haven't investigated them myself.
**What you expected to happen**:
The connection is closed properly by the server instead of packets being dropped. That way clients can fail fast and reestablish the connection.
If this is unavoidable, the exact values of when the control plane starts dropping packets should documented clearly somewhere so the Kubernetes Client Libraries and application developers can adjust their defaults or provide AKS-specific configuration.
**How to reproduce it (as minimally and precisely as possible)**:
Run this inside of the cluster, I'm using a `python:3.7.3-stretch` container.
```python
import time
import logging
from kubernetes import config, client
logging.basicConfig(
format='%(asctime)s %(levelname)-8s %(message)s',
level = logging.DEBUG)
config.load_incluster_config()
v1 = client.CoreV1Api()
logging.info('Calling 1st time')
v1.list_namespaced_pod('default')
logging.info('Sleeping 5 minutes')
time.sleep(300)
logging.info('Calling 2nd time')
# this call will timeout after 15 minutes
v1.list_namespaced_pod('default')
logging.info('OK')
```
**Environment**:
- Kubernetes version: v1.12.6
- Size of cluster: 1
This becomes especially noticeable when using watches, as the connection seems to remain open but stops receiving events. That connection is expected to be idle for many minutes between events. I've noticed it with the python client not inside and outside the cluster, connecting to the public master FQDN.
The basic nlb that uses the API has 4 Min idle timeout.
This is going to be changed in near future.
You might want to use init container with your deployment
Something like this:
initContainers:
- name: "sysctl"
image: "busybox:latest"
resources:
limits:
cpu: "10m"
memory: "8Mi"
requests:
cpu: "10m"
memory: "8Mi"
securityContext:
privileged: true
command:
- "/bin/sh"
- "-c"
- |
set -o errexit
set -o xtrace
sysctl -w net.ipv4.tcp_keepalive_time=180
sysctl -w net.ipv4.tcp_keepalive_intvl=180
sysctl -w net.ipv4.tcp_keepalive_probes=4
Hope it helps
This is directly related to the use of Azure Basic Load Balancers for AKS clusters. https://github.com/Azure/AKS/issues/643 and is on the public roadmap: https://github.com/Azure/AKS/projects/1
@jnoller Thanks for your reply. Glad to see the team is aware and has plans to remedy it.
I'll follow along at #643
Re-opening as this is separate from customers being able to deploy standard load balancers for their own services.
when this becomes GA; would this require a re-deployment of the AKS cluster? I would expect this to be an implicit change.
In case re-deployment is required, would enabling this need new commands in az cli/Terraform?
This issue has been automatically marked as stale because it has not had activity in 90 days. It will be closed if no further activity occurs. Thank you!
Hi, is this still being worked on? It's still something we're having to hack around.
Issue still active.
I can confirm this is an issue on AKS, I ran @itszakko job on AWS EKS and there is no timeout. This affects Airflow on K8S and Kubeflow deployments.
Azure Logs: https://github.com/maganaluis/k8s-api-job/blob/master/azure-aks.log
AWS EKS: https://github.com/maganaluis/k8s-api-job/blob/master/aws-eks.log
@itszakko is your workaround simply querying it every 3rd minute or so?@jluk @jnoller is this fixed with the GA?
This makes it hard to run services such as Airflow, Kubeflow and Prefect, where there often is long-lived process.
I believe this is the same issue highlighted here: https://github.com/Azure/AKS/issues/1877
I believe you are right @shooker
Do you know if a fix is under-way @shooker ?空Private Cluster API server silently closes connection
|!--
This issue tracker is a best-effort forum for users and customers to suggest features and report bugs.
If you are experiencing a service disruption when creating, upgrading, scaling, or deleting your cluster,
please open a support request with Azure support. Be sure to include your subscription id, resource group,
cluster name, and region. This information should __NOT__ be included in this issue due to its potential
sensitivity.
--|
|!--
Security issues should be reported to secure@microsoft.com and not via this public issue tracker.
--|
**What happened**: Attempts to use custom operator in private cluster to watch the cluster with long timeouts seem to lose connectivity with no connection reset being issued.
**What you expected to happen**: Connection reset be issued.
**How to reproduce it (as minimally and precisely as possible)**: Configure private cluster and use a custom operator to query the API server to watch the cluster and notice connection is silently dropped after 4 minutes.
Simple test is to run pod in private AKS cluster and exec in to its shell to run curl :
curl --no-keepalive --http1.1 -o/dev/null --cacert /run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" "https://${KUBERNETES_SERVICE_HOST}/api/v1/namespaces/default/pods?watch=1|timeoutSeconds=300"
In public cluster this connection is dropped/reset at 4 minutes but on private cluster this runs perpetually.
**Anything else we need to know?**:
**Environment**:
- Kubernetes version (use `kubectl version`): All
- Size of cluster (how many worker nodes are in the cluster?) N/A
- General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.) Custom operators
- Others: N/A
Hi erlucier, AKS bot here :wave:
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.
I might be just a bot, but I'm told my suggestions are normally quite good, as such:
1) If this case is urgent, please [open a Support Request](https://azure.microsoft.com/en-us/support/create-ticket/) so that our 24/7 support team may help you faster.
2) Please abide by the [AKS repo Guidelines](https://github.com/Azure/AKS#bug-reports-) and [Code of Conduct](https://github.com/Azure/AKS#code-of-conduct).
3) If you're having an issue, could it be described on the [AKS Troubleshooting guides](https://docs.microsoft.com/en-us/azure/aks/troubleshooting) or [AKS Diagnostics](https://docs.microsoft.com/en-us/azure/aks/concepts-diagnostics)?
4) Make sure your subscribed to the [AKS Release Notes](https://github.com/Azure/AKS/releases) to keep up to date with all that's new on AKS.
5) Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
6) If you have a question, do take a look at our [AKS FAQ](https://docs.microsoft.com/en-us/azure/aks/faq). We place the most common ones there!
Triage required from @Azure/aks-pm
Action required from @Azure/aks-pm
True class: both
Text with highlighted words
Odd text editor behavior in the code sheet.
|!-- FreeCodeCamp Issue Template --|
|!-- NOTE: ISSUES ARE NOT FOR CODE HELP - Ask for Help at https://gitter.im/FreeCodeCamp/Help --|
|!-- Please provide as much detail as possible for us to fix your issue --|
|!-- Remove any heading sections you did not fill out --|
#### Challenge Name
|!-- Insert link to challenge below --|
Give a Background Color to a Div Element
#### Issue Description
|!-- Describe below when the issue happens and how to reproduce it --|
On a new load of the page I click in the code area and start typing and the text appears one
row below where Iclicked in at. I also have some issues where there are certain parts of the code
text that I can'tclick in to or put my cursor in to.
#### Browser Information
|!-- Describe your workspace in which you are having issues--|
- Browser Name, Version: chrome Version 50.0.2661.75 m.
- Operating System: Windows 10
- Mobile, Desktop, or Tablet: Desktop
#### Screenshot
|!-- Add a screenshot of your issue --|
https://beem132-gmail.tinytake.com/sf/NjAzNTg1XzMwMDkzNjY
That's a link to a video showing the behavior in question I know this is the screen shot section but
the video is small and short and better at showing the issue, didn't have the ability to clip it but there's no new info after 30 seconds, hope it's useful.
空Cursor Skipping in Code Display
Challenge [Add Font Awesome Icons to all of our Buttons](https://www.freecodecamp.com/challenges/add-font-awesome-icons-to-all-of-our-buttons) has an issue.
User Agent is: |code|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36|/code|.
Please describe how to reproduce this issue, and include links to screenshots if possible.
**I'm not sure if this is just on my machine or not:** When i'm trying to access a certain part of my code, the cursor keeps skipping over code when i try to navigate to the target area ( both mouse and directional keys), and as such i can not edit it to progress.
My code:
``` html
|link href="http://fonts.googleapis.com/css?family=Lobster" rel="stylesheet" type="text/css"|
|style|
h2 {
font-family: Lobster, Monospace;
}
.thick-green-border {
border-color: green;
border-width: 10px;
border-style: solid;
border-radius: 50%;
}
|/style|
|div class="container-fluid"|
|div class="row"|
|div class="col-xs-8"|
|h2 class="text-primary text-center"|CatPhotoApp|/h2|
|/div|
|div class="col-xs-4"|
|a href="#"||img class="img-responsive thick-green-border" src="https://bit.ly/fcc-relaxing-cat"||/a|
|/div|
|/div|
|img src="http://bit.ly/fcc-running-cats" class="img-responsive"|
|div class="row"|
|div class="col-xs-4"|
|button class="btn btn-block btn-primary"||i class="fa fa-thumbs-up"||/i| Like|/button|
|/div|
|div class="col-xs-4"|
|button class="btn btn-block btn-info"|Info|/button|
|/div|
|div class="col-xs-4"|
|button class="btn btn-block btn-danger"|Delete|/button|
|/div|
|/div|
|p|Things cats |span class="text-danger"|love:|/span||/p|
|ul|
|li|cat nip|/li|
|li|laser pointers|/li|
|li|lasagna|/li|
|/ul|
|p|Top 3 things cats hate:|/p|
|ol|
|li|flea treatment|/li|
|li|thunder|/li|
|li|other cats|/li|
|/ol|
|form action="/submit-cat-photo"|
|label||input type="radio" name="indoor-outdoor"| Indoor|/label|
|label||input type="radio" name="indoor-outdoor"| Outdoor|/label|
|label||input type="checkbox" name="personality"| Loving|/label|
|label||input type="checkbox" name="personality"| Lazy|/label|
|label||input type="checkbox" name="personality"| Crazy|/label|
|input type="text" placeholder="cat photo URL" required|
|button type="submit"|Submit|/button|
|/form|
|/div|
```
I've tried your code in Windows/Mozilla environment and was not able to reproduce the workflow.
Seems to be related to issue #7528.
@Will-Campbell Thanks a lot for the issue. And apologies for the inconvenience.
I have been unable to reproduce this, but as seen others have been reproducing this as well. We should start looking into this.
Thanks @alistermada for pointing out the related issue.
Closing this in favor of #7528 . Please continue the discussion there.
Facing the same issue.